2023-12-15T16:40:33.086 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-15T16:40:33.086 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_lflores@teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --verbose --name lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi --archive /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262 -- /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262/orig.config.yaml 2023-12-15T16:40:33.138 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-15T16:40:33.222 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262 branch: wip-yuri10-testing-2023-12-12-1229 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7493262' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri10-testing-2023-12-12-1229 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri10-testing-2023-12-12-1229 sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 owner: scheduled_lflores@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9963 sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2023-12-12-1229 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 targets: smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC2/eNabySf0IuZlE+BHIHvM/dceKYYIFmCN2lkbZT9YqRA6QeS/zPOLyRDI83a0zpFobrC9ZbJzDK85o4PM83k= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJSe1/+NcQykE1SfbIJYGnBvdauZOKvJs4f8q5eTJzeX4G6JG/f5bVREqsRWL1z0Szf90F/G1grP+vyeGpIPDBk= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-15_16:16:51 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-15T16:40:33.222 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa; will attempt to use it 2023-12-15T16:40:33.223 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks 2023-12-15T16:40:33.223 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-15T16:40:33.225 INFO:teuthology.task.internal:Checking packages... 2023-12-15T16:40:33.249 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '021ac1670a21c19759080c6da90baaeb42e7d175' 2023-12-15T16:40:33.249 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-15T16:40:33.249 INFO:teuthology.packaging:ref: None 2023-12-15T16:40:33.249 INFO:teuthology.packaging:tag: None 2023-12-15T16:40:33.249 INFO:teuthology.packaging:branch: wip-yuri10-testing-2023-12-12-1229 2023-12-15T16:40:33.249 INFO:teuthology.packaging:sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-15T16:40:33.250 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri10-testing-2023-12-12-1229 2023-12-15T16:40:33.396 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-157.g021ac167 2023-12-15T16:40:33.397 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-15T16:40:33.406 INFO:teuthology.task.internal:no buildpackages task found 2023-12-15T16:40:33.406 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-15T16:40:33.428 INFO:teuthology.task.internal:Saving configuration 2023-12-15T16:40:33.444 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-15T16:40:33.450 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-15T16:40:33.475 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-15 16:35:31.136718', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC2/eNabySf0IuZlE+BHIHvM/dceKYYIFmCN2lkbZT9YqRA6QeS/zPOLyRDI83a0zpFobrC9ZbJzDK85o4PM83k='} 2023-12-15T16:40:33.499 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-15 16:35:31.134701', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJSe1/+NcQykE1SfbIJYGnBvdauZOKvJs4f8q5eTJzeX4G6JG/f5bVREqsRWL1z0Szf90F/G1grP+vyeGpIPDBk='} 2023-12-15T16:40:33.500 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-15T16:40:33.506 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-15T16:40:33.506 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-15T16:40:33.506 INFO:teuthology.run_tasks:Running task console_log... 2023-12-15T16:40:33.596 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feb189abe50>, signals=[15]) 2023-12-15T16:40:33.596 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-15T16:40:33.602 INFO:teuthology.task.internal:Opening connections... 2023-12-15T16:40:33.602 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2023-12-15T16:40:33.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:40:33.687 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2023-12-15T16:40:33.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:40:33.772 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-15T16:40:33.778 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2023-12-15T16:40:33.796 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2023-12-15T16:40:33.796 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2023-12-15T16:40:33.851 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-15T16:40:33.852 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-15T16:40:33.853 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2023-12-15T16:40:33.872 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2023-12-15T16:40:33.890 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2023-12-15T16:40:33.890 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2023-12-15T16:40:33.947 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2023-12-15T16:40:33.947 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2023-12-15T16:40:33.947 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2023-12-15T16:40:33.947 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2023-12-15T16:40:33.947 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2023-12-15T16:40:33.947 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2023-12-15T16:40:33.947 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-15T16:40:33.947 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2023-12-15T16:40:33.947 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2023-12-15T16:40:33.948 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-15T16:40:33.948 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2023-12-15T16:40:33.948 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-15T16:40:33.948 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-15T16:40:33.948 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-15T16:40:33.948 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2023-12-15T16:40:33.971 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-15T16:40:33.980 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-15T16:40:34.028 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-15T16:40:34.029 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2023-12-15T16:40:34.031 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2023-12-15T16:40:34.047 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-15T16:40:34.053 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-15T16:40:34.053 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2023-12-15T16:40:34.088 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2023-12-15T16:40:34.182 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-15T16:40:34.272 INFO:teuthology.run_tasks:Running task kernel... 2023-12-15T16:40:34.286 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-15T16:40:34.286 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-15T16:40:34.286 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-15T16:40:34.287 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-15T16:40:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T16:40:34.303 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-12-15T16:40:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T16:40:34.305 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2023-12-15T16:40:34.360 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-361.el9.x86_64 2023-12-15T16:40:34.360 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.14.0-361.el9.x86_64 2023-12-15T16:40:34.360 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2023-12-15T16:40:34.362 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-361.el9.x86_64 2023-12-15T16:40:34.362 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.14.0-361.el9.x86_64 2023-12-15T16:40:34.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2023-12-15T16:40:36.358 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2023-12-15T16:40:36.444 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - AppStream 18 MB/s | 6.4 MB 00:00 2023-12-15T16:40:37.683 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2023-12-15T16:40:38.044 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2023-12-15T16:40:39.584 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 7.9 MB 00:01 2023-12-15T16:40:40.301 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 7.9 MB 00:01 2023-12-15T16:40:41.994 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 28 MB/s | 18 MB 00:00 2023-12-15T16:40:46.257 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 18 MB 00:05 2023-12-15T16:40:46.638 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 5.8 MB 00:01 2023-12-15T16:40:48.078 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2023-12-15T16:40:49.441 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2023-12-15T16:40:50.016 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 14 MB/s | 5.8 MB 00:00 2023-12-15T16:40:51.467 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 15 kB 00:00 2023-12-15T16:40:54.105 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2023-12-15T16:40:55.295 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 20 MB 00:03 2023-12-15T16:40:56.279 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-15T16:40:56.279 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-15T16:40:56.374 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:40:56.378 INFO:teuthology.orchestra.run.smithi192.stdout:Install 4 Packages 2023-12-15T16:40:56.379 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:40:56.379 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 96 M 2023-12-15T16:40:56.379 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 121 M 2023-12-15T16:40:56.379 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:40:57.223 INFO:teuthology.orchestra.run.smithi192.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 9.9 MB/s | 5.6 MB 00:00 2023-12-15T16:40:58.073 INFO:teuthology.orchestra.run.smithi192.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 14 MB/s | 20 MB 00:01 2023-12-15T16:40:59.098 INFO:teuthology.orchestra.run.smithi192.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 16 MB/s | 38 MB 00:02 2023-12-15T16:40:59.449 INFO:teuthology.orchestra.run.smithi192.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 15 MB/s | 32 MB 00:02 2023-12-15T16:40:59.449 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:40:59.449 INFO:teuthology.orchestra.run.smithi192.stdout:Total 31 MB/s | 96 MB 00:03 2023-12-15T16:41:00.151 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2023-12-15T16:41:00.306 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:41:00.650 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:41:00.650 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:41:01.934 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:41:01.935 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:41:02.266 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-15T16:41:02.266 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-15T16:41:02.370 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:41:02.382 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:41:02.382 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2023-12-15T16:41:02.382 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:41:02.382 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:41:02.382 INFO:teuthology.orchestra.run.smithi069.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-15T16:41:02.382 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-12-15T16:41:02.382 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-15T16:41:02.382 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-15T16:41:02.382 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-15T16:41:02.382 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:41:02.383 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:41:02.383 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:41:02.383 INFO:teuthology.orchestra.run.smithi069.stdout:Install 4 Packages 2023-12-15T16:41:02.383 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:41:02.383 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 96 M 2023-12-15T16:41:02.383 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 121 M 2023-12-15T16:41:02.383 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:41:02.794 INFO:teuthology.orchestra.run.smithi069.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 15 MB/s | 5.6 MB 00:00 2023-12-15T16:41:03.502 INFO:teuthology.orchestra.run.smithi069.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 19 MB/s | 20 MB 00:01 2023-12-15T16:41:04.344 INFO:teuthology.orchestra.run.smithi069.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 20 MB/s | 38 MB 00:01 2023-12-15T16:41:04.720 INFO:teuthology.orchestra.run.smithi069.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 17 MB/s | 32 MB 00:01 2023-12-15T16:41:04.720 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:41:04.720 INFO:teuthology.orchestra.run.smithi069.stdout:Total 41 MB/s | 96 MB 00:02 2023-12-15T16:41:04.859 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:41:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-15T16:41:05.565 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:41:05.634 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T16:41:05.936 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:41:05.936 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:41:07.228 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:41:07.228 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:41:08.352 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T16:41:08.527 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T16:41:10.125 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:41:10.791 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-15T16:41:10.928 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T16:41:13.148 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T16:41:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:13.654 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T16:41:13.821 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T16:41:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:18.405 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T16:41:18.595 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:23.236 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:46.631 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-15T16:41:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T16:41:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T16:41:47.445 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:47.445 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:41:47.446 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:41:47.446 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-15T16:41:47.446 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-15T16:41:47.446 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-15T16:41:47.446 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-15T16:41:47.446 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:41:47.446 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:41:47.939 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2023-12-15T16:41:48.079 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-15T16:41:48.080 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-15T16:41:48.080 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-15T16:41:48.080 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-15T16:41:48.080 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-15T16:41:48.080 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-15T16:41:48.080 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2023-12-15T16:41:48.080 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2023-12-15T16:41:48.198 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-15T16:41:48.198 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-15T16:41:48.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo rpm -qi grub2-tools 2023-12-15T16:41:48.231 INFO:teuthology.orchestra.run.smithi192.stdout:Name : grub2-tools 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:Epoch : 1 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:Version : 2.06 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:Release : 68.el9 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: x86_64 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:Group : Unspecified 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:Size : 8264254 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:License : GPLv3+ 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-15T16:41:48.232 INFO:teuthology.orchestra.run.smithi192.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout:Packager : builder@centos.org 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout:Vendor : CentOS 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout:Summary : Support tools for GRUB. 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout:Description : 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout:hardware devices. 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:41:48.233 INFO:teuthology.orchestra.run.smithi192.stdout:This subpackage provides tools for support of all platforms. 2023-12-15T16:41:48.234 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-15T16:41:48.234 INFO:teuthology.task.kernel:Updating grub on smithi192 to boot 5.14.0-391.el9.x86_64 2023-12-15T16:41:48.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-15T16:41:48.844 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2023-12-15T16:41:50.214 INFO:teuthology.orchestra.run.smithi192.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-15T16:41:50.251 INFO:teuthology.orchestra.run.smithi192.stderr:done 2023-12-15T16:41:50.253 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2023-12-15T16:41:50.269 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.uOqfV1oG6G 2023-12-15T16:41:50.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uOqfV1oG6G 2023-12-15T16:41:50.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.uOqfV1oG6G 2023-12-15T16:41:50.477 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.uOqfV1oG6G is 6KB 2023-12-15T16:41:50.525 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.uOqfV1oG6G 2023-12-15T16:41:50.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo /bin/ls /boot/loader/entries || true 2023-12-15T16:41:50.609 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-15T16:41:50.609 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-15T16:41:50.609 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-15T16:41:50.609 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-15T16:41:50.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-15T16:41:50.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2023-12-15T16:41:52.154 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:52.195 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:52.645 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:52.645 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-15T16:41:52.645 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T16:41:52.645 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T16:41:53.102 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T16:41:53.102 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:41:53.102 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:41:53.102 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-15T16:41:53.102 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-15T16:41:53.102 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-15T16:41:53.102 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-15T16:41:53.102 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:41:53.102 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:41:53.628 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2023-12-15T16:41:53.763 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-15T16:41:53.764 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-15T16:41:53.764 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-15T16:41:53.764 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-15T16:41:53.764 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-15T16:41:53.764 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-15T16:41:53.764 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2023-12-15T16:41:53.764 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2023-12-15T16:41:53.881 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-15T16:41:53.882 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-15T16:41:53.882 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2023-12-15T16:41:53.925 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2023-12-15T16:41:53.925 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2023-12-15T16:41:53.925 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2023-12-15T16:41:53.925 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2023-12-15T16:41:53.925 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:41:53.926 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-15T16:41:53.927 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-15T16:41:53.927 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-15T16:41:53.927 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2023-12-15T16:41:53.927 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:41:53.927 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2023-12-15T16:41:53.928 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-15T16:41:53.928 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 5.14.0-391.el9.x86_64 2023-12-15T16:41:53.928 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-15T16:41:54.528 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2023-12-15T16:41:56.018 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-15T16:41:56.047 INFO:teuthology.orchestra.run.smithi069.stderr:done 2023-12-15T16:41:56.053 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-12-15T16:41:56.068 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.WG46WfLT2F 2023-12-15T16:41:56.068 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WG46WfLT2F 2023-12-15T16:41:56.136 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.WG46WfLT2F 2023-12-15T16:41:56.280 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.WG46WfLT2F is 6KB 2023-12-15T16:41:56.328 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.WG46WfLT2F 2023-12-15T16:41:56.342 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2023-12-15T16:41:56.409 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-15T16:41:56.409 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-15T16:41:56.410 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-15T16:41:56.410 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-15T16:41:56.410 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-15T16:41:56.560 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2023-12-15T16:42:20.801 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-15T16:42:20.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-12-15T16:42:20.802 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:42:26.568 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-15T16:42:26.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-12-15T16:42:26.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:42:39.250 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-12-15T16:42:45.141 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2023-12-15T16:42:48.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-12-15T16:42:48.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:42:51.313 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-12-15T16:42:54.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-12-15T16:42:54.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:42:57.201 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2023-12-15T16:43:03.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-12-15T16:43:03.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:43:06.386 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-12-15T16:43:09.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-12-15T16:43:09.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:43:21.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-12-15T16:43:21.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:43:24.465 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-12-15T16:43:41.035 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-12-15T16:43:41.462 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-12-15T16:43:41.462 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-391.el9.x86_64"... 2023-12-15T16:43:41.462 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-12-15T16:43:41.476 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-391.el9.x86_64 2023-12-15T16:43:41.477 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-15T16:43:41.477 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-15T16:43:41.477 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-15T16:43:42.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-12-15T16:43:42.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:43:42.478 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-15T16:43:42.478 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-15T16:43:42.605 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2023-12-15T16:43:42.713 DEBUG:teuthology.parallel:result is None 2023-12-15T16:43:42.922 DEBUG:teuthology.orchestra.run.smithi192:> true 2023-12-15T16:43:43.278 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-12-15T16:43:43.278 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-391.el9.x86_64"... 2023-12-15T16:43:43.278 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2023-12-15T16:43:43.294 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-391.el9.x86_64 2023-12-15T16:43:43.294 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-15T16:43:43.294 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-15T16:43:43.294 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-15T16:43:44.295 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-15T16:43:44.296 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-15T16:43:44.361 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2023-12-15T16:43:44.403 DEBUG:teuthology.parallel:result is None 2023-12-15T16:43:44.404 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-15T16:43:44.410 INFO:teuthology.task.internal:Creating test directory... 2023-12-15T16:43:44.411 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-15T16:43:44.413 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-15T16:43:44.432 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-15T16:43:44.455 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-15T16:43:44.488 INFO:teuthology.task.internal:Creating archive directory... 2023-12-15T16:43:44.488 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-15T16:43:44.491 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-15T16:43:44.554 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-15T16:43:44.561 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-15T16:43:44.561 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-15T16:43:44.564 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-15T16:43:44.590 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-15T16:43:44.598 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-15T16:43:44.602 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-15T16:43:44.612 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-15T16:43:44.613 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-15T16:43:44.621 INFO:teuthology.task.internal:Configuring sudo... 2023-12-15T16:43:44.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-15T16:43:44.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-15T16:43:44.683 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-15T16:43:44.691 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-15T16:43:44.691 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-15T16:43:44.721 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-15T16:43:44.738 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-15T16:43:44.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-15T16:43:44.880 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-15T16:43:44.966 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-15T16:43:45.011 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T16:43:45.011 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-15T16:43:45.082 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-15T16:43:45.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-15T16:43:45.177 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-15T16:43:45.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-15T16:43:45.305 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T16:43:45.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-15T16:43:45.373 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2023-12-15T16:43:45.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2023-12-15T16:43:45.420 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-15T16:43:45.472 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-15T16:43:45.969 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-15T16:43:45.980 INFO:teuthology.task.internal:Starting timer... 2023-12-15T16:43:45.980 INFO:teuthology.run_tasks:Running task pcp... 2023-12-15T16:43:46.008 INFO:teuthology.run_tasks:Running task selinux... 2023-12-15T16:43:46.041 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2023-12-15T16:43:46.110 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2023-12-15T16:43:46.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2023-12-15T16:43:46.178 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2023-12-15T16:43:46.180 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-15T16:43:46.180 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2023-12-15T16:43:46.206 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2023-12-15T16:43:46.206 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2023-12-15T16:43:46.230 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2023-12-15T16:43:46.230 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi069.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2023-12-15T16:43:46.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-15T16:43:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T16:43:46.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-15T16:43:46.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T16:43:46.306 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-15T16:43:46.306 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2023-12-15T16:43:46.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2023-12-15T16:43:46.407 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-15T16:43:46.418 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-15T16:43:46.418 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-15T16:43:46.438 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-15T16:43:46.439 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi069.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-15T16:49:20.181 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2023-12-15T16:49:20.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-12-15T16:49:20.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:49:20.265 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-12-15T16:49:20.331 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-12-15T16:49:20.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-12-15T16:49:20.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T16:49:20.420 DEBUG:teuthology.orchestra.run.smithi192:> true 2023-12-15T16:49:20.488 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-12-15T16:49:20.488 INFO:teuthology.run_tasks:Running task clock... 2023-12-15T16:49:20.498 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-15T16:49:20.499 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-15T16:49:20.499 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-15T16:49:20.502 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-15T16:49:20.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-15T16:49:20.529 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-15T16:49:20.545 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-15T16:49:20.561 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-15T16:49:20.574 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2023-12-15T16:49:20.578 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-15T16:49:20.587 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2023-12-15T16:49:20.604 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-15T16:49:20.608 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2023-12-15T16:49:20.620 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-15T16:49:20.621 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2023-12-15T16:49:20.638 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-15T16:49:20.654 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-15T16:49:20.662 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2023-12-15T16:49:20.665 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-15T16:49:20.665 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2023-12-15T16:49:20.665 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T16:49:20.666 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T16:49:20.666 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T16:49:20.666 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T16:49:20.697 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2023-12-15T16:49:20.701 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-15T16:49:20.701 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2023-12-15T16:49:20.701 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T16:49:20.701 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T16:49:20.701 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T16:49:20.701 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T16:49:20.701 INFO:teuthology.run_tasks:Running task install... 2023-12-15T16:49:20.710 DEBUG:teuthology.task.install:project ceph 2023-12-15T16:49:20.710 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175'}} 2023-12-15T16:49:20.710 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175'} 2023-12-15T16:49:20.710 INFO:teuthology.task.install:Using flavor: default 2023-12-15T16:49:20.716 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-12-15T16:49:20.716 INFO:teuthology.task.install:extra packages: [] 2023-12-15T16:49:20.717 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175', 'tag': None, 'wait_for_package': False} 2023-12-15T16:49:20.718 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-15T16:49:20.719 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175', 'tag': None, 'wait_for_package': False} 2023-12-15T16:49:20.720 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-15T16:49:20.866 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/ 2023-12-15T16:49:20.866 INFO:teuthology.task.install.rpm:Package version is 19.0.0-157.g021ac167 2023-12-15T16:49:20.869 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/ 2023-12-15T16:49:20.870 INFO:teuthology.task.install.rpm:Package version is 19.0.0-157.g021ac167 2023-12-15T16:49:20.974 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-15T16:49:20.974 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T16:49:20.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-15T16:49:21.002 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-15T16:49:21.002 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/021ac1670a21c19759080c6da90baaeb42e7d175/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-15T16:49:21.034 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-15T16:49:21.034 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T16:49:21.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-15T16:49:21.063 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-15T16:49:21.063 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/021ac1670a21c19759080c6da90baaeb42e7d175/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-15T16:49:21.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-15T16:49:21.131 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-15T16:49:21.150 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-15T16:49:21.211 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-15T16:49:21.217 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2023-12-15T16:49:21.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2023-12-15T16:49:21.277 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2023-12-15T16:49:21.279 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2023-12-15T16:49:21.458 INFO:teuthology.orchestra.run.smithi192.stdout:73 files removed 2023-12-15T16:49:21.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw 2023-12-15T16:49:21.547 INFO:teuthology.orchestra.run.smithi069.stdout:73 files removed 2023-12-15T16:49:21.573 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw 2023-12-15T16:49:22.189 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 260 kB/s | 78 kB 00:00 2023-12-15T16:49:22.198 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 197 kB/s | 78 kB 00:00 2023-12-15T16:49:22.641 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 24 kB/s | 9.5 kB 00:00 2023-12-15T16:49:22.715 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 20 kB/s | 9.5 kB 00:00 2023-12-15T16:49:22.991 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2023-12-15T16:49:23.057 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2023-12-15T16:49:23.443 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 18 MB/s | 6.4 MB 00:00 2023-12-15T16:49:23.469 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-15T16:49:24.786 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2023-12-15T16:49:24.870 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2023-12-15T16:49:25.851 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 7.9 MB 00:00 2023-12-15T16:49:27.092 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 7.9 MB 00:01 2023-12-15T16:49:29.243 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 16 MB/s | 18 MB 00:01 2023-12-15T16:49:33.277 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 5.8 MB 00:00 2023-12-15T16:49:34.778 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2023-12-15T16:49:43.323 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 2.3 MB/s | 20 MB 00:08 2023-12-15T16:49:48.028 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-12-15T16:49:50.471 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:49:50.474 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:49:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-12-15T16:49:50.474 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:49:50.474 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:49:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:19.0.0-157.g021ac167.el9 ceph 8.6 M 2023-12-15T16:49:50.475 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2023-12-15T16:49:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.3 M 2023-12-15T16:49:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.0 M 2023-12-15T16:49:50.475 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-12-15T16:49:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-15T16:49:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:19.0.0-157.g021ac167.el9 ceph 5.2 M 2023-12-15T16:49:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:19.0.0-157.g021ac167.el9 ceph 19 M 2023-12-15T16:49:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:19.0.0-157.g021ac167.el9 ceph 24 k 2023-12-15T16:49:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-15T16:49:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-15T16:49:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-15T16:49:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 719 k 2023-12-15T16:49:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-15T16:49:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-15T16:49:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-15T16:49:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:19.0.0-157.g021ac167.el9 ceph 476 k 2023-12-15T16:49:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-15T16:49:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 4.8 M 2023-12-15T16:49:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-15T16:49:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-15T16:49:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-15T16:49:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse x86_64 2:19.0.0-157.g021ac167.el9 ceph 45 k 2023-12-15T16:49:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:19.0.0-157.g021ac167.el9 ceph 129 k 2023-12-15T16:49:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:19.0.0-157.g021ac167.el9 ceph 161 k 2023-12-15T16:49:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-15T16:49:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:19.0.0-157.g021ac167.el9 ceph 321 k 2023-12-15T16:49:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:19.0.0-157.g021ac167.el9 ceph 298 k 2023-12-15T16:49:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:19.0.0-157.g021ac167.el9 ceph 99 k 2023-12-15T16:49:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-15T16:49:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-15T16:49:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:49:50.478 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:49:50.478 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:49:50.478 INFO:teuthology.orchestra.run.smithi069.stdout:Install 27 Packages 2023-12-15T16:49:50.478 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 2 Packages 2023-12-15T16:49:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:49:50.479 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 55 M 2023-12-15T16:49:50.479 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:49:51.212 INFO:teuthology.orchestra.run.smithi069.stdout:(1/29): ceph-radosgw-19.0.0-157.g021ac167.el9.x 13 MB/s | 8.6 MB 00:00 2023-12-15T16:49:51.265 INFO:teuthology.orchestra.run.smithi069.stdout:(2/29): ceph-selinux-19.0.0-157.g021ac167.el9.x 483 kB/s | 24 kB 00:00 2023-12-15T16:49:51.346 INFO:teuthology.orchestra.run.smithi069.stdout:(3/29): libcephfs2-19.0.0-157.g021ac167.el9.x86 8.4 MB/s | 719 kB 00:00 2023-12-15T16:49:51.405 INFO:teuthology.orchestra.run.smithi069.stdout:(4/29): libradosstriper1-19.0.0-157.g021ac167.e 8.0 MB/s | 476 kB 00:00 2023-12-15T16:49:51.722 INFO:teuthology.orchestra.run.smithi069.stdout:(5/29): librgw2-19.0.0-157.g021ac167.el9.x86_64 15 MB/s | 4.8 MB 00:00 2023-12-15T16:49:51.763 INFO:teuthology.orchestra.run.smithi069.stdout:(6/29): python3-ceph-argparse-19.0.0-157.g021ac 1.1 MB/s | 45 kB 00:00 2023-12-15T16:49:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:(7/29): python3-ceph-common-19.0.0-157.g021ac16 1.9 MB/s | 129 kB 00:00 2023-12-15T16:49:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:(8/29): ceph-base-19.0.0-157.g021ac167.el9.x86_ 3.8 MB/s | 5.2 MB 00:01 2023-12-15T16:49:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:(9/29): python3-cephfs-19.0.0-157.g021ac167.el9 1.7 MB/s | 161 kB 00:00 2023-12-15T16:49:51.973 INFO:teuthology.orchestra.run.smithi069.stdout:(10/29): python3-rados-19.0.0-157.g021ac167.el9 4.2 MB/s | 321 kB 00:00 2023-12-15T16:49:51.998 INFO:teuthology.orchestra.run.smithi069.stdout:(11/29): python3-rbd-19.0.0-157.g021ac167.el9.x 3.9 MB/s | 298 kB 00:00 2023-12-15T16:49:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:(12/29): python3-rgw-19.0.0-157.g021ac167.el9.x 1.2 MB/s | 99 kB 00:00 2023-12-15T16:49:52.099 INFO:teuthology.orchestra.run.smithi069.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2023-12-15T16:49:52.124 INFO:teuthology.orchestra.run.smithi069.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2023-12-15T16:49:52.141 INFO:teuthology.orchestra.run.smithi069.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2023-12-15T16:49:52.183 INFO:teuthology.orchestra.run.smithi069.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-12-15T16:49:52.208 INFO:teuthology.orchestra.run.smithi069.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2023-12-15T16:49:52.234 INFO:teuthology.orchestra.run.smithi069.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2023-12-15T16:49:52.509 INFO:teuthology.orchestra.run.smithi069.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2023-12-15T16:49:52.626 INFO:teuthology.orchestra.run.smithi069.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2023-12-15T16:49:52.851 INFO:teuthology.orchestra.run.smithi069.stdout:(21/29): ceph-common-19.0.0-157.g021ac167.el9.x 8.2 MB/s | 19 MB 00:02 2023-12-15T16:49:52.877 INFO:teuthology.orchestra.run.smithi069.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 194 kB/s | 49 kB 00:00 2023-12-15T16:49:52.994 INFO:teuthology.orchestra.run.smithi069.stdout:(23/29): libarrow-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2023-12-15T16:49:53.035 INFO:teuthology.orchestra.run.smithi069.stdout:(24/29): libunwind-1.6.2-1.el9.x86_64.rpm 367 kB/s | 67 kB 00:00 2023-12-15T16:49:53.069 INFO:teuthology.orchestra.run.smithi069.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2023-12-15T16:49:53.094 INFO:teuthology.orchestra.run.smithi069.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2023-12-15T16:49:53.278 INFO:teuthology.orchestra.run.smithi069.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2023-12-15T16:49:53.387 INFO:teuthology.orchestra.run.smithi069.stdout:(28/29): librbd1-19.0.0-157.g021ac167.el9.x86_6 10 MB/s | 3.0 MB 00:00 2023-12-15T16:49:53.596 INFO:teuthology.orchestra.run.smithi069.stdout:(29/29): librados2-19.0.0-157.g021ac167.el9.x86 6.3 MB/s | 3.3 MB 00:00 2023-12-15T16:49:53.599 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:49:53.599 INFO:teuthology.orchestra.run.smithi069.stdout:Total 18 MB/s | 55 MB 00:03 2023-12-15T16:49:53.600 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:49:53.632 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:49:53.632 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:49:54.084 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:49:54.085 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:49:55.430 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:49:55.520 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-15T16:49:55.566 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-15T16:49:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-15T16:49:55.944 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-15T16:49:56.008 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:19.0.0-157.g021ac167.el9.x86_64 5/31 2023-12-15T16:49:56.083 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:19.0.0-157.g021ac167.el9.x86_64 5/31 2023-12-15T16:49:56.183 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 6/31 2023-12-15T16:49:56.210 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 7/31 2023-12-15T16:49:56.255 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 7/31 2023-12-15T16:49:56.292 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-15T16:49:56.336 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:19.0.0-157.g021ac167.el9.x 9/31 2023-12-15T16:49:56.403 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 10/31 2023-12-15T16:49:56.428 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_64 11/31 2023-12-15T16:49:56.494 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_64 11/31 2023-12-15T16:49:56.601 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-15T16:49:56.636 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-15T16:49:56.677 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-15T16:49:56.709 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-15T16:49:56.737 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-15T16:49:56.770 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-15T16:49:57.070 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-15T16:49:57.100 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:19.0.0-157.g021ac167.el9.x86_64 19/31 2023-12-15T16:49:57.199 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:19.0.0-157.g021ac167.el9.x86_64 19/31 2023-12-15T16:49:57.229 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 20/31 2023-12-15T16:49:57.274 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-15T16:49:57.697 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-15T16:49:57.727 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:19.0.0-157.g021ac167.el9.x86_64 23/31 2023-12-15T16:49:57.782 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:19.0.0-157.g021ac167.el9.x86_64 23/31 2023-12-15T16:49:57.858 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 24/31 2023-12-15T16:49:57.880 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:19.0.0-157.g021ac167.el9.x86 25/31 2023-12-15T16:50:00.011 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 26/31 2023-12-15T16:50:00.051 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 26/31 2023-12-15T16:50:00.643 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 26/31 2023-12-15T16:50:00.679 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 27/31 2023-12-15T16:50:00.749 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 27/31 2023-12-15T16:50:00.749 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-15T16:50:00.749 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-15T16:50:00.749 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:00.797 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 28/31 2023-12-15T16:50:10.241 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 28/31 2023-12-15T16:50:10.242 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2023-12-15T16:50:10.242 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2023-12-15T16:50:10.242 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2023-12-15T16:50:10.242 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2023-12-15T16:50:10.242 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2023-12-15T16:50:10.242 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /root 2023-12-15T16:50:10.242 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2023-12-15T16:50:10.242 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:11.070 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 29/31 2023-12-15T16:50:11.094 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 29/31 2023-12-15T16:50:11.095 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:50:11.095 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-15T16:50:11.095 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-15T16:50:11.095 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-15T16:50:11.095 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:11.096 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-15T16:50:11.181 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-15T16:50:11.181 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T16:50:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T16:50:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 1/31 2023-12-15T16:50:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 2/31 2023-12-15T16:50:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 3/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 4/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 5/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_64 6/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:19.0.0-157.g021ac167.el9.x86_64 7/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:19.0.0-157.g021ac167.el9.x 8/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:19.0.0-157.g021ac167.el9.x86 9/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 10/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 11/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 12/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 13/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-15T16:50:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:19.0.0-157.g021ac167.el9.x86_64 28/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-15T16:50:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:19.0.0-157.g021ac167.el9.x86_64 30/31 2023-12-15T16:50:13.290 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T16:50:13.290 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:13.290 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2023-12-15T16:50:13.290 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-15T16:50:13.291 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: re2-1:20211101-3.el9.x86_64 2023-12-15T16:50:13.292 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-15T16:50:13.293 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:13.293 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:50:13.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-test 2023-12-15T16:50:14.092 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:26 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:50:14.236 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:50:14.236 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:14.236 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-12-15T16:50:14.236 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:14.236 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:50:14.236 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:19.0.0-157.g021ac167.el9 ceph 41 M 2023-12-15T16:50:14.237 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-12-15T16:50:14.237 INFO:teuthology.orchestra.run.smithi069.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-15T16:50:14.237 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:19.0.0-157.g021ac167.el9 ceph 163 k 2023-12-15T16:50:14.237 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-15T16:50:14.237 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-15T16:50:14.237 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-15T16:50:14.237 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:14.237 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:50:14.237 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:14.237 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2023-12-15T16:50:14.238 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:14.238 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 42 M 2023-12-15T16:50:14.238 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 173 M 2023-12-15T16:50:14.239 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:50:14.387 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-12-15T16:50:14.429 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-15T16:50:14.471 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-15T16:50:14.513 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-15T16:50:14.538 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): libcephsqlite-19.0.0-157.g021ac167.el9.x 546 kB/s | 163 kB 00:00 2023-12-15T16:50:16.280 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): ceph-test-19.0.0-157.g021ac167.el9.x86_6 20 MB/s | 41 MB 00:02 2023-12-15T16:50:16.283 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:50:16.283 INFO:teuthology.orchestra.run.smithi069.stdout:Total 21 MB/s | 42 MB 00:02 2023-12-15T16:50:16.284 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:50:16.303 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:50:16.304 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:50:16.656 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:50:16.657 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:50:17.192 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:50:17.261 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-15T16:50:17.300 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-15T16:50:17.329 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-15T16:50:17.363 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-15T16:50:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 5/6 2023-12-15T16:50:21.231 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 5/6 2023-12-15T16:50:21.308 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 6/6 2023-12-15T16:50:23.367 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 6/6 2023-12-15T16:50:23.367 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 1/6 2023-12-15T16:50:23.367 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 2/6 2023-12-15T16:50:23.367 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-15T16:50:23.367 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-15T16:50:23.367 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-15T16:50:23.734 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-15T16:50:23.735 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:23.735 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:50:23.735 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:23.735 INFO:teuthology.orchestra.run.smithi069.stdout: jq-1.6-16.el9.x86_64 2023-12-15T16:50:23.735 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:23.735 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-15T16:50:23.735 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-15T16:50:23.735 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-15T16:50:23.735 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:23.735 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:50:24.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph 2023-12-15T16:50:24.568 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:36 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:50:24.723 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:50:24.726 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-12-15T16:50:24.726 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-12-15T16:50:24.726 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-12-15T16:50:24.726 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:50:24.726 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:19.0.0-157.g021ac167.el9 ceph 6.5 k 2023-12-15T16:50:24.726 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-12-15T16:50:24.726 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:19.0.0-157.g021ac167.el9 ceph 2.1 M 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:19.0.0-157.g021ac167.el9 ceph 1.5 M 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 243 k 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:19.0.0-157.g021ac167.el9 ceph 4.4 M 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:19.0.0-157.g021ac167.el9 ceph 18 M 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-15T16:50:24.727 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-15T16:50:24.728 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-15T16:50:24.729 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-15T16:50:24.729 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-15T16:50:24.729 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-15T16:50:24.729 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:24.729 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:50:24.729 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2023-12-15T16:50:24.729 INFO:teuthology.orchestra.run.smithi069.stdout:Install 35 Packages 2023-12-15T16:50:24.729 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:24.730 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 30 M 2023-12-15T16:50:24.730 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 111 M 2023-12-15T16:50:24.730 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:50:24.948 INFO:teuthology.orchestra.run.smithi069.stdout:(1/35): ceph-19.0.0-157.g021ac167.el9.x86_64.rp 40 kB/s | 6.5 kB 00:00 2023-12-15T16:50:25.215 INFO:teuthology.orchestra.run.smithi069.stdout:(2/35): ceph-mgr-19.0.0-157.g021ac167.el9.x86_6 3.4 MB/s | 1.5 MB 00:00 2023-12-15T16:50:25.316 INFO:teuthology.orchestra.run.smithi069.stdout:(3/35): ceph-mon-19.0.0-157.g021ac167.el9.x86_6 12 MB/s | 4.4 MB 00:00 2023-12-15T16:50:25.383 INFO:teuthology.orchestra.run.smithi069.stdout:(4/35): ceph-mgr-modules-core-19.0.0-157.g021ac 3.6 MB/s | 243 kB 00:00 2023-12-15T16:50:25.500 INFO:teuthology.orchestra.run.smithi069.stdout:(5/35): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-12-15T16:50:25.541 INFO:teuthology.orchestra.run.smithi069.stdout:(6/35): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-12-15T16:50:25.575 INFO:teuthology.orchestra.run.smithi069.stdout:(7/35): python3-markupsafe-1.1.1-12.el9.x86_64. 944 kB/s | 32 kB 00:00 2023-12-15T16:50:25.609 INFO:teuthology.orchestra.run.smithi069.stdout:(8/35): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-15T16:50:25.642 INFO:teuthology.orchestra.run.smithi069.stdout:(9/35): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-15T16:50:25.684 INFO:teuthology.orchestra.run.smithi069.stdout:(10/35): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-12-15T16:50:25.743 INFO:teuthology.orchestra.run.smithi069.stdout:(11/35): python3-cryptography-36.0.1-4.el9.x86_ 20 MB/s | 1.1 MB 00:00 2023-12-15T16:50:25.777 INFO:teuthology.orchestra.run.smithi069.stdout:(12/35): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-12-15T16:50:25.810 INFO:teuthology.orchestra.run.smithi069.stdout:(13/35): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-12-15T16:50:25.869 INFO:teuthology.orchestra.run.smithi069.stdout:(14/35): ceph-mds-19.0.0-157.g021ac167.el9.x86_ 1.9 MB/s | 2.1 MB 00:01 2023-12-15T16:50:25.894 INFO:teuthology.orchestra.run.smithi069.stdout:(15/35): python3-requests-2.25.1-7.el9.noarch.r 1.3 MB/s | 113 kB 00:00 2023-12-15T16:50:25.919 INFO:teuthology.orchestra.run.smithi069.stdout:(16/35): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-12-15T16:50:26.011 INFO:teuthology.orchestra.run.smithi069.stdout:(17/35): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 473 kB/s | 43 kB 00:00 2023-12-15T16:50:26.195 INFO:teuthology.orchestra.run.smithi069.stdout:(18/35): ceph-osd-19.0.0-157.g021ac167.el9.x86_ 18 MB/s | 18 MB 00:00 2023-12-15T16:50:26.212 INFO:teuthology.orchestra.run.smithi069.stdout:(19/35): luarocks-3.9.2-1.el9.noarch.rpm 476 kB/s | 151 kB 00:00 2023-12-15T16:50:26.254 INFO:teuthology.orchestra.run.smithi069.stdout:(20/35): python3-cheroot-8.6.0-4.el9.noarch.rpm 709 kB/s | 172 kB 00:00 2023-12-15T16:50:26.296 INFO:teuthology.orchestra.run.smithi069.stdout:(21/35): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2023-12-15T16:50:26.321 INFO:teuthology.orchestra.run.smithi069.stdout:(22/35): python3-jaraco-classes-3.2.1-5.el9.noa 709 kB/s | 18 kB 00:00 2023-12-15T16:50:26.338 INFO:teuthology.orchestra.run.smithi069.stdout:(23/35): python3-jaraco-collections-3.0.0-8.el9 554 kB/s | 23 kB 00:00 2023-12-15T16:50:26.355 INFO:teuthology.orchestra.run.smithi069.stdout:(24/35): python3-jaraco-functools-3.5.0-2.el9.n 578 kB/s | 19 kB 00:00 2023-12-15T16:50:26.372 INFO:teuthology.orchestra.run.smithi069.stdout:(25/35): python3-jaraco-text-3.2.0-6.el9.noarch 590 kB/s | 20 kB 00:00 2023-12-15T16:50:26.397 INFO:teuthology.orchestra.run.smithi069.stdout:(26/35): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2023-12-15T16:50:26.423 INFO:teuthology.orchestra.run.smithi069.stdout:(27/35): python3-logutils-0.3.5-21.el9.noarch.r 688 kB/s | 46 kB 00:00 2023-12-15T16:50:26.448 INFO:teuthology.orchestra.run.smithi069.stdout:(28/35): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-12-15T16:50:26.473 INFO:teuthology.orchestra.run.smithi069.stdout:(29/35): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2023-12-15T16:50:26.507 INFO:teuthology.orchestra.run.smithi069.stdout:(30/35): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2023-12-15T16:50:26.532 INFO:teuthology.orchestra.run.smithi069.stdout:(31/35): python3-tempora-5.0.0-2.el9.noarch.rpm 610 kB/s | 36 kB 00:00 2023-12-15T16:50:26.557 INFO:teuthology.orchestra.run.smithi069.stdout:(32/35): python3-pyOpenSSL-21.0.0-1.el9.noarch. 825 kB/s | 90 kB 00:00 2023-12-15T16:50:26.608 INFO:teuthology.orchestra.run.smithi069.stdout:(33/35): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-12-15T16:50:26.633 INFO:teuthology.orchestra.run.smithi069.stdout:(34/35): python3-zc-lockfile-2.0-10.el9.noarch. 263 kB/s | 20 kB 00:00 2023-12-15T16:50:26.692 INFO:teuthology.orchestra.run.smithi069.stdout:(35/35): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2023-12-15T16:50:26.695 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:50:26.695 INFO:teuthology.orchestra.run.smithi069.stdout:Total 15 MB/s | 30 MB 00:01 2023-12-15T16:50:26.696 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:50:26.736 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:50:26.736 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:50:27.142 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:50:27.142 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:50:28.491 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:50:28.545 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/35 2023-12-15T16:50:28.577 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/35 2023-12-15T16:50:28.609 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/35 2023-12-15T16:50:28.701 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/35 2023-12-15T16:50:28.771 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/35 2023-12-15T16:50:28.812 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/35 2023-12-15T16:50:28.884 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/35 2023-12-15T16:50:28.937 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/35 2023-12-15T16:50:28.976 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/35 2023-12-15T16:50:29.025 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/35 2023-12-15T16:50:29.080 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/35 2023-12-15T16:50:29.263 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-15T16:50:29.314 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-15T16:50:29.377 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/35 2023-12-15T16:50:29.418 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/35 2023-12-15T16:50:29.450 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/35 2023-12-15T16:50:29.526 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/35 2023-12-15T16:50:29.569 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/35 2023-12-15T16:50:29.603 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/35 2023-12-15T16:50:29.643 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/35 2023-12-15T16:50:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/35 2023-12-15T16:50:29.712 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/35 2023-12-15T16:50:29.811 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/35 2023-12-15T16:50:29.851 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/35 2023-12-15T16:50:29.927 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/35 2023-12-15T16:50:30.069 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/35 2023-12-15T16:50:30.186 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/35 2023-12-15T16:50:30.346 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9.n 28/35 2023-12-15T16:50:30.373 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 29/35 2023-12-15T16:50:30.397 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 29/35 2023-12-15T16:50:30.398 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:50:30.398 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-15T16:50:30.398 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-15T16:50:30.398 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-15T16:50:30.398 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:30.480 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-5.4.4-4.el9.x86_64 30/35 2023-12-15T16:50:32.059 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : luarocks-3.9.2-1.el9.noarch 31/35 2023-12-15T16:50:32.085 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 32/35 2023-12-15T16:50:32.113 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 32/35 2023-12-15T16:50:32.114 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:50:32.114 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-15T16:50:32.114 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-15T16:50:32.114 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-15T16:50:32.114 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:32.561 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 33/35 2023-12-15T16:50:32.584 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 33/35 2023-12-15T16:50:32.584 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:50:32.584 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-15T16:50:32.584 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-15T16:50:32.584 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-15T16:50:32.584 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 34/35 2023-12-15T16:50:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 34/35 2023-12-15T16:50:32.810 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:50:32.810 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-15T16:50:32.810 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-15T16:50:32.810 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-15T16:50:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:32.875 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:19.0.0-157.g021ac167.el9.x86_64 35/35 2023-12-15T16:50:34.786 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-2:19.0.0-157.g021ac167.el9.x86_64 35/35 2023-12-15T16:50:34.786 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:19.0.0-157.g021ac167.el9.x86_64 1/35 2023-12-15T16:50:34.786 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 2/35 2023-12-15T16:50:34.786 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 3/35 2023-12-15T16:50:34.786 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 4/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 5/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9.n 6/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 18/35 2023-12-15T16:50:34.787 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 19/35 2023-12-15T16:50:34.788 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 20/35 2023-12-15T16:50:34.788 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 21/35 2023-12-15T16:50:34.788 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 22/35 2023-12-15T16:50:34.788 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 23/35 2023-12-15T16:50:34.788 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 24/35 2023-12-15T16:50:34.788 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 25/35 2023-12-15T16:50:34.788 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 26/35 2023-12-15T16:50:34.788 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 27/35 2023-12-15T16:50:34.789 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 28/35 2023-12-15T16:50:34.789 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 29/35 2023-12-15T16:50:34.789 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 30/35 2023-12-15T16:50:34.789 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 31/35 2023-12-15T16:50:34.790 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 32/35 2023-12-15T16:50:34.790 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 33/35 2023-12-15T16:50:34.790 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 34/35 2023-12-15T16:50:35.877 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 35/35 2023-12-15T16:50:35.877 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: lua-5.4.4-4.el9.x86_64 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-15T16:50:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-15T16:50:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply-3.11-14.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:35.880 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:50:36.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base 2023-12-15T16:50:36.779 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:48 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:50:36.871 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:50:36.920 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:50:36.920 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-12-15T16:50:36.921 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:50:36.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install cephadm 2023-12-15T16:50:37.532 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:49 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:50:37.674 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:50:37.675 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:37.675 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-12-15T16:50:37.675 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:37.675 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:50:37.675 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 803 k 2023-12-15T16:50:37.675 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:37.675 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:50:37.675 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:37.675 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:50:37.676 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:37.676 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 803 k 2023-12-15T16:50:37.676 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 818 k 2023-12-15T16:50:37.676 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:50:37.993 INFO:teuthology.orchestra.run.smithi069.stdout:cephadm-19.0.0-157.g021ac167.el9.noarch.rpm 2.5 MB/s | 803 kB 00:00 2023-12-15T16:50:37.994 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:50:37.994 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.5 MB/s | 803 kB 00:00 2023-12-15T16:50:37.994 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:50:38.001 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:50:38.001 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:50:38.005 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:50:38.005 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:50:38.079 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:50:38.511 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:50:38.605 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:50:38.982 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:50:39.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:50:39.307 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:39.307 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:50:39.307 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:50:39.307 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:39.307 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:50:39.495 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-immutable-object-cache 2023-12-15T16:50:40.061 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:52 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:50:40.210 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:50:40.211 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:40.211 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:50:40.211 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:40.211 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:50:40.211 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-157.g021ac167.el9 ceph 141 k 2023-12-15T16:50:40.211 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:40.211 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:50:40.212 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:40.212 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:50:40.212 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:40.212 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 141 k 2023-12-15T16:50:40.212 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 395 k 2023-12-15T16:50:40.212 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:50:40.416 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-immutable-object-cache-19.0.0-157.g021ac16 694 kB/s | 141 kB 00:00 2023-12-15T16:50:40.416 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:50:40.416 INFO:teuthology.orchestra.run.smithi069.stdout:Total 688 kB/s | 141 kB 00:00 2023-12-15T16:50:40.416 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:50:40.424 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:50:40.424 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:50:40.475 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:50:40.475 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:50:40.616 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:50:40.677 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-157.g021ac167.e 1/1 2023-12-15T16:50:40.699 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-157.g021ac167.e 1/1 2023-12-15T16:50:40.699 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:50:40.699 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-15T16:50:40.699 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:41.655 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-157.g021ac167.e 1/1 2023-12-15T16:50:41.656 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:41.656 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:50:41.656 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:50:41.656 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:41.656 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:50:41.868 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr 2023-12-15T16:50:41.961 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 249 kB/s | 18 MB 01:15 2023-12-15T16:50:42.429 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:54 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:50:42.521 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:50:42.570 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:50:42.571 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-12-15T16:50:42.571 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:50:42.634 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-dashboard 2023-12-15T16:50:43.187 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:55 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:50:43.330 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:50:43.331 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:43.331 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-12-15T16:50:43.331 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:43.331 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:50:43.331 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 1.8 M 2023-12-15T16:50:43.331 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-12-15T16:50:43.331 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 28 k 2023-12-15T16:50:43.331 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 15 k 2023-12-15T16:50:43.332 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-15T16:50:43.332 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-15T16:50:43.332 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-15T16:50:43.332 INFO:teuthology.orchestra.run.smithi069.stdout:Installing weak dependencies: 2023-12-15T16:50:43.332 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-15T16:50:43.332 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:43.332 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:50:43.332 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:50:43.332 INFO:teuthology.orchestra.run.smithi069.stdout:Install 7 Packages 2023-12-15T16:50:43.332 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:43.333 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 2.1 M 2023-12-15T16:50:43.333 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 8.7 M 2023-12-15T16:50:43.333 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:50:43.694 INFO:teuthology.orchestra.run.smithi069.stdout:(1/7): ceph-grafana-dashboards-19.0.0-157.g021a 133 kB/s | 28 kB 00:00 2023-12-15T16:50:43.736 INFO:teuthology.orchestra.run.smithi069.stdout:(2/7): ceph-prometheus-alerts-19.0.0-157.g021ac 60 kB/s | 15 kB 00:00 2023-12-15T16:50:43.944 INFO:teuthology.orchestra.run.smithi069.stdout:(3/7): ceph-mgr-dashboard-19.0.0-157.g021ac167. 3.8 MB/s | 1.8 MB 00:00 2023-12-15T16:50:44.162 INFO:teuthology.orchestra.run.smithi069.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 19 kB/s | 9.0 kB 00:00 2023-12-15T16:50:44.320 INFO:teuthology.orchestra.run.smithi069.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 70 kB/s | 41 kB 00:00 2023-12-15T16:50:44.371 INFO:teuthology.orchestra.run.smithi069.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 81 kB/s | 31 kB 00:00 2023-12-15T16:50:44.737 INFO:teuthology.orchestra.run.smithi069.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 328 kB/s | 188 kB 00:00 2023-12-15T16:50:44.742 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:50:44.742 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.5 MB/s | 2.1 MB 00:01 2023-12-15T16:50:44.742 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:50:44.751 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:50:44.751 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:50:44.821 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:50:44.821 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:50:45.117 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:50:45.189 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-15T16:50:45.222 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-15T16:50:45.268 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-15T16:50:45.301 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-15T16:50:45.331 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el9.no 5/7 2023-12-15T16:50:45.569 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-157.g021ac167.el9.n 6/7 2023-12-15T16:50:45.601 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.noarch 7/7 2023-12-15T16:50:46.200 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.noarch 7/7 2023-12-15T16:50:46.201 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-157.g021ac167.el9.n 1/7 2023-12-15T16:50:46.201 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.noarch 2/7 2023-12-15T16:50:46.201 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el9.no 3/7 2023-12-15T16:50:46.201 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-15T16:50:46.201 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-15T16:50:46.201 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-15T16:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 5.8 MB 00:00 2023-12-15T16:50:46.514 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-15T16:50:46.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:46.514 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:50:46.514 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:50:46.514 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:50:46.514 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:50:46.514 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-15T16:50:46.514 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-15T16:50:46.514 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-15T16:50:46.514 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-15T16:50:46.515 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:46.515 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:50:46.739 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-15T16:50:47.292 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:59 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:50:47.435 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:50:47.436 INFO:teuthology.orchestra.run.smithi069.stdout:=========================================================================================== 2023-12-15T16:50:47.436 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-12-15T16:50:47.436 INFO:teuthology.orchestra.run.smithi069.stdout:=========================================================================================== 2023-12-15T16:50:47.436 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:50:47.436 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 7.4 M 2023-12-15T16:50:47.436 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-12-15T16:50:47.436 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout:=========================================================================================== 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout:Install 12 Packages 2023-12-15T16:50:47.437 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:47.438 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 37 M 2023-12-15T16:50:47.438 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 209 M 2023-12-15T16:50:47.438 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:50:47.853 INFO:teuthology.orchestra.run.smithi069.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 32 kB 00:00 2023-12-15T16:50:47.886 INFO:teuthology.orchestra.run.smithi069.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2023-12-15T16:50:47.944 INFO:teuthology.orchestra.run.smithi069.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2023-12-15T16:50:47.970 INFO:teuthology.orchestra.run.smithi069.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 420 kB/s | 34 kB 00:00 2023-12-15T16:50:48.078 INFO:teuthology.orchestra.run.smithi069.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 35 MB/s | 4.7 MB 00:00 2023-12-15T16:50:48.087 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 15 kB 00:00 2023-12-15T16:50:48.154 INFO:teuthology.orchestra.run.smithi069.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2023-12-15T16:50:48.179 INFO:teuthology.orchestra.run.smithi069.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-12-15T16:50:48.238 INFO:teuthology.orchestra.run.smithi069.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 14 MB/s | 803 kB 00:00 2023-12-15T16:50:48.396 INFO:teuthology.orchestra.run.smithi069.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 65 MB/s | 16 MB 00:00 2023-12-15T16:50:48.421 INFO:teuthology.orchestra.run.smithi069.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-12-15T16:50:48.663 INFO:teuthology.orchestra.run.smithi069.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 922 kB/s | 245 kB 00:00 2023-12-15T16:50:48.772 INFO:teuthology.orchestra.run.smithi069.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 7.1 MB/s | 7.4 MB 00:01 2023-12-15T16:50:48.774 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:50:48.775 INFO:teuthology.orchestra.run.smithi069.stdout:Total 28 MB/s | 37 MB 00:01 2023-12-15T16:50:48.788 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:50:48.804 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 33 MB/s | 20 MB 00:00 2023-12-15T16:50:48.812 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:50:48.812 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:50:49.188 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:50:49.188 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:50:49.882 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:50:49.953 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-15T16:50:49.984 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-15T16:50:50.076 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-15T16:50:50.129 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-15T16:50:50.305 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-15T16:50:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-15T16:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-15T16:50:51.075 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-15T16:50:51.173 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-15T16:50:52.379 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-15T16:50:53.385 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-15T16:50:53.411 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-157.g021ac1 12/12 2023-12-15T16:50:53.495 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2023-12-15T16:50:55.963 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:50:55.971 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:50:55.971 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-12-15T16:50:55.971 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:50:55.971 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:50:55.971 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:19.0.0-157.g021ac167.el9 ceph 8.6 M 2023-12-15T16:50:55.971 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.3 M 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.0 M 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:19.0.0-157.g021ac167.el9 ceph 5.2 M 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:19.0.0-157.g021ac167.el9 ceph 19 M 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:19.0.0-157.g021ac167.el9 ceph 24 k 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 719 k 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-15T16:50:55.972 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:19.0.0-157.g021ac167.el9 ceph 476 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 4.8 M 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:19.0.0-157.g021ac167.el9 ceph 45 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:19.0.0-157.g021ac167.el9 ceph 129 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:19.0.0-157.g021ac167.el9 ceph 161 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:19.0.0-157.g021ac167.el9 ceph 321 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:19.0.0-157.g021ac167.el9 ceph 298 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:19.0.0-157.g021ac167.el9 ceph 99 k 2023-12-15T16:50:55.973 INFO:teuthology.orchestra.run.smithi192.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-15T16:50:55.974 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-15T16:50:55.974 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:50:55.974 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:50:55.974 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:50:55.974 INFO:teuthology.orchestra.run.smithi192.stdout:Install 27 Packages 2023-12-15T16:50:55.974 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 2 Packages 2023-12-15T16:50:55.974 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:50:55.974 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 55 M 2023-12-15T16:50:55.974 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-157.g021ac1 12/12 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-157.g021ac1 1/12 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-15T16:50:55.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-15T16:50:56.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-15T16:50:56.416 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:56.416 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:50:56.416 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:56.417 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:50:56.769 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-rook 2023-12-15T16:50:57.013 INFO:teuthology.orchestra.run.smithi192.stdout:(1/29): ceph-radosgw-19.0.0-157.g021ac167.el9.x 8.9 MB/s | 8.6 MB 00:00 2023-12-15T16:50:57.055 INFO:teuthology.orchestra.run.smithi192.stdout:(2/29): ceph-selinux-19.0.0-157.g021ac167.el9.x 581 kB/s | 24 kB 00:00 2023-12-15T16:50:57.189 INFO:teuthology.orchestra.run.smithi192.stdout:(3/29): libcephfs2-19.0.0-157.g021ac167.el9.x86 5.3 MB/s | 719 kB 00:00 2023-12-15T16:50:57.272 INFO:teuthology.orchestra.run.smithi192.stdout:(4/29): libradosstriper1-19.0.0-157.g021ac167.e 5.6 MB/s | 476 kB 00:00 2023-12-15T16:50:57.333 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:09 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:50:57.476 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:50:57.477 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2023-12-15T16:50:57.477 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-12-15T16:50:57.477 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2023-12-15T16:50:57.477 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:50:57.477 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 50 k 2023-12-15T16:50:57.477 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-12-15T16:50:57.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-15T16:50:57.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-15T16:50:57.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-15T16:50:57.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-15T16:50:57.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-15T16:50:57.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-15T16:50:57.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-15T16:50:57.478 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-15T16:50:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-15T16:50:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-15T16:50:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-15T16:50:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-15T16:50:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:57.479 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:50:57.479 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2023-12-15T16:50:57.479 INFO:teuthology.orchestra.run.smithi069.stdout:Install 13 Packages 2023-12-15T16:50:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:50:57.480 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 2.0 M 2023-12-15T16:50:57.480 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 26 M 2023-12-15T16:50:57.480 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:50:57.644 INFO:teuthology.orchestra.run.smithi069.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 217 kB/s | 24 kB 00:00 2023-12-15T16:50:57.661 INFO:teuthology.orchestra.run.smithi069.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 140 kB/s | 18 kB 00:00 2023-12-15T16:50:57.689 INFO:teuthology.orchestra.run.smithi192.stdout:(5/29): ceph-common-19.0.0-157.g021ac167.el9.x8 12 MB/s | 19 MB 00:01 2023-12-15T16:50:57.703 INFO:teuthology.orchestra.run.smithi069.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2023-12-15T16:50:57.736 INFO:teuthology.orchestra.run.smithi069.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-12-15T16:50:57.748 INFO:teuthology.orchestra.run.smithi192.stdout:(6/29): python3-ceph-argparse-19.0.0-157.g021ac 775 kB/s | 45 kB 00:00 2023-12-15T16:50:57.762 INFO:teuthology.orchestra.run.smithi069.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2023-12-15T16:50:57.779 INFO:teuthology.orchestra.run.smithi069.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2023-12-15T16:50:57.815 INFO:teuthology.orchestra.run.smithi192.stdout:(7/29): librgw2-19.0.0-157.g021ac167.el9.x86_64 8.8 MB/s | 4.8 MB 00:00 2023-12-15T16:50:57.837 INFO:teuthology.orchestra.run.smithi069.stdout:(7/13): ceph-mgr-rook-19.0.0-157.g021ac167.el9. 164 kB/s | 50 kB 00:00 2023-12-15T16:50:57.840 INFO:teuthology.orchestra.run.smithi192.stdout:(8/29): python3-ceph-common-19.0.0-157.g021ac16 1.4 MB/s | 129 kB 00:00 2023-12-15T16:50:57.866 INFO:teuthology.orchestra.run.smithi192.stdout:(9/29): python3-cephfs-19.0.0-157.g021ac167.el9 3.1 MB/s | 161 kB 00:00 2023-12-15T16:50:57.902 INFO:teuthology.orchestra.run.smithi192.stdout:(10/29): python3-rados-19.0.0-157.g021ac167.el9 5.3 MB/s | 321 kB 00:00 2023-12-15T16:50:57.933 INFO:teuthology.orchestra.run.smithi192.stdout:(11/29): python3-rbd-19.0.0-157.g021ac167.el9.x 4.3 MB/s | 298 kB 00:00 2023-12-15T16:50:57.958 INFO:teuthology.orchestra.run.smithi192.stdout:(12/29): python3-rgw-19.0.0-157.g021ac167.el9.x 1.7 MB/s | 99 kB 00:00 2023-12-15T16:50:58.004 INFO:teuthology.orchestra.run.smithi069.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 66 kB/s | 15 kB 00:00 2023-12-15T16:50:58.021 INFO:teuthology.orchestra.run.smithi069.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 124 kB/s | 32 kB 00:00 2023-12-15T16:50:58.051 INFO:teuthology.orchestra.run.smithi192.stdout:(13/29): ceph-base-19.0.0-157.g021ac167.el9.x86 2.6 MB/s | 5.2 MB 00:02 2023-12-15T16:50:58.118 INFO:teuthology.orchestra.run.smithi192.stdout:(14/29): boost-program-options-1.75.0-8.el9.x86 575 kB/s | 106 kB 00:00 2023-12-15T16:50:58.134 INFO:teuthology.orchestra.run.smithi192.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 756 kB/s | 44 kB 00:00 2023-12-15T16:50:58.168 INFO:teuthology.orchestra.run.smithi192.stdout:(16/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 858 kB/s | 159 kB 00:00 2023-12-15T16:50:58.230 INFO:teuthology.orchestra.run.smithi069.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 503 kB/s | 197 kB 00:00 2023-12-15T16:50:58.255 INFO:teuthology.orchestra.run.smithi069.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2023-12-15T16:50:58.260 INFO:teuthology.orchestra.run.smithi192.stdout:(17/29): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 661 kB 00:00 2023-12-15T16:50:58.285 INFO:teuthology.orchestra.run.smithi192.stdout:(18/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 282 kB 00:00 2023-12-15T16:50:58.302 INFO:teuthology.orchestra.run.smithi192.stdout:(19/29): python3-prettytable-0.7.2-27.el9.noarc 308 kB/s | 41 kB 00:00 2023-12-15T16:50:58.330 INFO:teuthology.orchestra.run.smithi069.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 893 kB/s | 90 kB 00:00 2023-12-15T16:50:58.644 INFO:teuthology.orchestra.run.smithi192.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 76 kB/s | 26 kB 00:00 2023-12-15T16:50:58.853 INFO:teuthology.orchestra.run.smithi192.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2023-12-15T16:50:58.855 INFO:teuthology.orchestra.run.smithi069.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.2 MB/s | 1.0 MB 00:00 2023-12-15T16:50:58.857 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:50:58.858 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2023-12-15T16:50:58.858 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:50:58.872 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:50:58.872 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:50:58.878 INFO:teuthology.orchestra.run.smithi192.stdout:(22/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 500 kB/s | 309 kB 00:00 2023-12-15T16:50:58.976 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:50:58.977 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:50:58.995 INFO:teuthology.orchestra.run.smithi192.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 475 kB/s | 67 kB 00:00 2023-12-15T16:50:59.162 INFO:teuthology.orchestra.run.smithi192.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2023-12-15T16:50:59.404 INFO:teuthology.orchestra.run.smithi192.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.6 MB/s | 838 kB 00:00 2023-12-15T16:50:59.498 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:50:59.651 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-15T16:50:59.705 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-15T16:50:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-15T16:50:59.790 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-15T16:50:59.824 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-15T16:50:59.915 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-15T16:51:00.025 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-15T16:51:00.075 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-15T16:51:00.245 INFO:teuthology.orchestra.run.smithi192.stdout:(26/29): librados2-19.0.0-157.g021ac167.el9.x86 3.9 MB/s | 3.3 MB 00:00 2023-12-15T16:51:00.421 INFO:teuthology.orchestra.run.smithi192.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 1.3 MB/s | 1.6 MB 00:01 2023-12-15T16:51:00.679 INFO:teuthology.orchestra.run.smithi192.stdout:(28/29): librbd1-19.0.0-157.g021ac167.el9.x86_6 7.0 MB/s | 3.0 MB 00:00 2023-12-15T16:51:00.797 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-15T16:51:00.847 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-15T16:51:00.881 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-15T16:51:00.928 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-15T16:51:00.951 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 13/13 2023-12-15T16:51:01.604 INFO:teuthology.orchestra.run.smithi192.stdout:(29/29): libarrow-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 4.4 MB 00:03 2023-12-15T16:51:01.612 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:01.612 INFO:teuthology.orchestra.run.smithi192.stdout:Total 9.7 MB/s | 55 MB 00:05 2023-12-15T16:51:01.612 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:51:01.648 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:51:01.648 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:51:01.765 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 13/13 2023-12-15T16:51:01.765 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 1/13 2023-12-15T16:51:01.765 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-15T16:51:01.766 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-15T16:51:01.766 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-15T16:51:01.766 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-15T16:51:01.766 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-15T16:51:01.766 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-15T16:51:01.766 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-15T16:51:01.766 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-15T16:51:01.766 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-15T16:51:01.766 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-15T16:51:01.766 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-15T16:51:02.115 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:51:02.115 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:51:02.140 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-15T16:51:02.140 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-15T16:51:02.141 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-15T16:51:02.142 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-15T16:51:02.142 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-15T16:51:02.142 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:02.142 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:02.390 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-cephadm 2023-12-15T16:51:02.939 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:14 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:03.083 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:03.083 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2023-12-15T16:51:03.084 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-12-15T16:51:03.084 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2023-12-15T16:51:03.084 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:51:03.084 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 133 k 2023-12-15T16:51:03.084 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-12-15T16:51:03.084 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-15T16:51:03.084 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-15T16:51:03.084 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-15T16:51:03.084 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-15T16:51:03.084 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-15T16:51:03.084 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:03.085 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:51:03.085 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2023-12-15T16:51:03.085 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2023-12-15T16:51:03.085 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:03.085 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 6.8 M 2023-12-15T16:51:03.086 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 33 M 2023-12-15T16:51:03.086 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:51:03.490 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-12-15T16:51:03.497 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:51:03.577 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-15T16:51:03.609 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-15T16:51:03.632 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-12-15T16:51:03.676 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-15T16:51:03.682 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): ceph-mgr-cephadm-19.0.0-157.g021ac167.el 386 kB/s | 133 kB 00:00 2023-12-15T16:51:03.974 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 170 kB/s | 58 kB 00:00 2023-12-15T16:51:03.997 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-15T16:51:04.025 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 160 kB/s | 51 kB 00:00 2023-12-15T16:51:04.062 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:19.0.0-157.g021ac167.el9.x86_64 5/31 2023-12-15T16:51:04.153 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:19.0.0-157.g021ac167.el9.x86_64 5/31 2023-12-15T16:51:04.166 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 812 kB/s | 548 kB 00:00 2023-12-15T16:51:04.170 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:04.170 INFO:teuthology.orchestra.run.smithi069.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2023-12-15T16:51:04.170 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:51:04.179 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:51:04.179 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:51:04.241 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 6/31 2023-12-15T16:51:04.268 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:51:04.268 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:51:04.272 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 7/31 2023-12-15T16:51:04.318 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 7/31 2023-12-15T16:51:04.347 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-15T16:51:04.396 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:19.0.0-157.g021ac167.el9.x 9/31 2023-12-15T16:51:04.457 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 10/31 2023-12-15T16:51:04.482 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_64 11/31 2023-12-15T16:51:04.548 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_64 11/31 2023-12-15T16:51:04.564 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:51:04.654 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-15T16:51:04.690 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-15T16:51:04.728 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-15T16:51:04.730 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-15T16:51:04.763 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-15T16:51:04.776 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-15T16:51:04.792 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-15T16:51:04.825 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-15T16:51:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-15T16:51:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:19.0.0-157.g021ac167.el9.x86_64 19/31 2023-12-15T16:51:05.269 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:19.0.0-157.g021ac167.el9.x86_64 19/31 2023-12-15T16:51:05.292 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-15T16:51:05.300 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 20/31 2023-12-15T16:51:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-15T16:51:05.370 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-15T16:51:05.438 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-15T16:51:05.463 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noarch 6/6 2023-12-15T16:51:05.763 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-15T16:51:05.790 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:19.0.0-157.g021ac167.el9.x86_64 23/31 2023-12-15T16:51:05.861 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:19.0.0-157.g021ac167.el9.x86_64 23/31 2023-12-15T16:51:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 24/31 2023-12-15T16:51:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:19.0.0-157.g021ac167.el9.x86 25/31 2023-12-15T16:51:06.237 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noarch 6/6 2023-12-15T16:51:06.237 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noarch 1/6 2023-12-15T16:51:06.237 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-15T16:51:06.237 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-15T16:51:06.237 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-15T16:51:06.237 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-15T16:51:06.578 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-15T16:51:06.578 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:06.579 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:51:06.579 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:51:06.579 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-15T16:51:06.579 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-15T16:51:06.579 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-15T16:51:06.579 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-15T16:51:06.579 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-15T16:51:06.579 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:06.579 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:06.837 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-fuse 2023-12-15T16:51:07.394 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:19 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:07.538 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:07.540 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:07.540 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-12-15T16:51:07.540 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:19.0.0-157.g021ac167.el9 ceph 859 k 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 938 k 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.7 M 2023-12-15T16:51:07.541 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:51:07.659 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 661 kB/s | 79 kB 00:00 2023-12-15T16:51:07.859 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-fuse-19.0.0-157.g021ac167.el9.x86_6 2.6 MB/s | 859 kB 00:00 2023-12-15T16:51:07.860 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:07.861 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.9 MB/s | 938 kB 00:00 2023-12-15T16:51:07.861 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:51:07.881 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:51:07.882 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:51:07.958 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:51:07.958 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:51:08.117 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 26/31 2023-12-15T16:51:08.147 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 26/31 2023-12-15T16:51:08.152 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:51:08.292 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-15T16:51:08.359 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 2/2 2023-12-15T16:51:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 26/31 2023-12-15T16:51:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 27/31 2023-12-15T16:51:08.849 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 27/31 2023-12-15T16:51:08.849 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-15T16:51:08.849 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-15T16:51:08.850 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:08.885 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 28/31 2023-12-15T16:51:09.047 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 2/2 2023-12-15T16:51:09.047 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 1/2 2023-12-15T16:51:09.361 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-15T16:51:09.361 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:09.361 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:51:09.361 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-15T16:51:09.361 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:09.361 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:09.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-volume 2023-12-15T16:51:10.186 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:22 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:10.336 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:10.336 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 262 k 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 262 k 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.2 M 2023-12-15T16:51:10.337 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:51:10.556 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-volume-19.0.0-157.g021ac167.el9.noarch.rpm 1.2 MB/s | 262 kB 00:00 2023-12-15T16:51:10.556 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:10.556 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.2 MB/s | 262 kB 00:00 2023-12-15T16:51:10.557 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:51:10.564 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:51:10.565 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:51:10.601 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:51:10.601 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:51:10.811 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:51:10.883 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:51:10.898 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:51:10.898 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:51:10.898 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-15T16:51:10.898 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:51:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:11.942 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:51:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:51:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:11.942 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:12.219 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados-devel 2023-12-15T16:51:12.804 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:24 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:12.948 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:19.0.0-157.g021ac167.el9 ceph 126 k 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 126 k 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 456 k 2023-12-15T16:51:12.949 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:51:13.171 INFO:teuthology.orchestra.run.smithi069.stdout:librados-devel-19.0.0-157.g021ac167.el9.x86_64. 572 kB/s | 126 kB 00:00 2023-12-15T16:51:13.172 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:13.172 INFO:teuthology.orchestra.run.smithi069.stdout:Total 569 kB/s | 126 kB 00:00 2023-12-15T16:51:13.172 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:51:13.177 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:51:13.177 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:51:13.233 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:51:13.233 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:51:13.377 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:51:13.478 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:13.912 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:14.211 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:14.211 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:14.211 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:51:14.211 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:14.211 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:14.211 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:14.487 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs2 2023-12-15T16:51:15.043 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:27 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:15.140 INFO:teuthology.orchestra.run.smithi069.stdout:Package libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:51:15.190 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:15.191 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-12-15T16:51:15.191 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:15.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs-devel 2023-12-15T16:51:15.810 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:27 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:15.954 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:15.955 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:15.955 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:51:15.955 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:15.955 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:51:15.955 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:19.0.0-157.g021ac167.el9 ceph 31 k 2023-12-15T16:51:15.955 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:15.955 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:51:15.955 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:15.955 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:51:15.955 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:15.955 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 31 k 2023-12-15T16:51:15.956 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 139 k 2023-12-15T16:51:15.956 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:51:16.200 INFO:teuthology.orchestra.run.smithi069.stdout:libcephfs-devel-19.0.0-157.g021ac167.el9.x86_64 128 kB/s | 31 kB 00:00 2023-12-15T16:51:16.201 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:16.201 INFO:teuthology.orchestra.run.smithi069.stdout:Total 127 kB/s | 31 kB 00:00 2023-12-15T16:51:16.201 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:51:16.204 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:51:16.204 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:51:16.230 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:51:16.230 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:51:16.310 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:51:16.422 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:16.808 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:17.077 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:17.077 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:17.077 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:51:17.077 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:17.077 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:17.077 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:17.245 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados2 2023-12-15T16:51:17.803 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:29 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:17.896 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:51:17.946 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:17.947 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-12-15T16:51:17.947 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:18.025 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librbd1 2023-12-15T16:51:18.497 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 28/31 2023-12-15T16:51:18.497 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2023-12-15T16:51:18.497 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2023-12-15T16:51:18.497 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2023-12-15T16:51:18.497 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2023-12-15T16:51:18.497 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2023-12-15T16:51:18.497 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /root 2023-12-15T16:51:18.498 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2023-12-15T16:51:18.498 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:18.579 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:30 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:18.674 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:51:18.724 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:18.725 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-12-15T16:51:18.725 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:18.791 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rados 2023-12-15T16:51:19.316 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 29/31 2023-12-15T16:51:19.336 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:31 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:19.340 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 29/31 2023-12-15T16:51:19.340 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:51:19.340 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-15T16:51:19.340 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-15T16:51:19.340 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-15T16:51:19.340 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:19.342 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-15T16:51:19.427 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-15T16:51:19.427 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T16:51:19.429 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:51:19.479 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:19.480 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-12-15T16:51:19.480 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:19.548 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rgw 2023-12-15T16:51:20.097 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:32 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:20.191 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:51:20.242 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:20.242 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-12-15T16:51:20.243 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:20.306 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-cephfs 2023-12-15T16:51:20.625 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T16:51:20.625 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 1/31 2023-12-15T16:51:20.625 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 2/31 2023-12-15T16:51:20.625 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 3/31 2023-12-15T16:51:20.625 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 4/31 2023-12-15T16:51:20.625 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 5/31 2023-12-15T16:51:20.625 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_64 6/31 2023-12-15T16:51:20.625 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:19.0.0-157.g021ac167.el9.x86_64 7/31 2023-12-15T16:51:20.625 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:19.0.0-157.g021ac167.el9.x 8/31 2023-12-15T16:51:20.625 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:19.0.0-157.g021ac167.el9.x86 9/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 10/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 11/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 12/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 13/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-15T16:51:20.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-15T16:51:20.627 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-15T16:51:20.627 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-15T16:51:20.627 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-15T16:51:20.627 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:19.0.0-157.g021ac167.el9.x86_64 28/31 2023-12-15T16:51:20.627 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-15T16:51:20.627 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:19.0.0-157.g021ac167.el9.x86_64 30/31 2023-12-15T16:51:20.858 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:32 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:20.951 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:51:21.001 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:21.002 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-12-15T16:51:21.002 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:21.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rbd 2023-12-15T16:51:21.623 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:33 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:21.716 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:51:21.750 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-15T16:51:21.751 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-15T16:51:21.752 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.753 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.753 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:21.753 INFO:teuthology.orchestra.run.smithi192.stdout: re2-1:20211101-3.el9.x86_64 2023-12-15T16:51:21.753 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-15T16:51:21.753 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:21.753 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:51:21.766 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:21.767 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-12-15T16:51:21.767 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:21.835 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-fuse 2023-12-15T16:51:22.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test 2023-12-15T16:51:22.388 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:34 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:22.534 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:19.0.0-157.g021ac167.el9 ceph 85 k 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 85 k 2023-12-15T16:51:22.535 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 231 k 2023-12-15T16:51:22.536 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:51:22.618 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:29 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:51:22.736 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-fuse-19.0.0-157.g021ac167.el9.x86_64.rpm 424 kB/s | 85 kB 00:00 2023-12-15T16:51:22.737 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:22.737 INFO:teuthology.orchestra.run.smithi069.stdout:Total 421 kB/s | 85 kB 00:00 2023-12-15T16:51:22.737 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:51:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:51:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:51:22.761 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:51:22.762 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:22.762 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-12-15T16:51:22.762 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:22.762 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:51:22.762 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:19.0.0-157.g021ac167.el9 ceph 41 M 2023-12-15T16:51:22.762 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-12-15T16:51:22.762 INFO:teuthology.orchestra.run.smithi192.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-15T16:51:22.762 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:19.0.0-157.g021ac167.el9 ceph 163 k 2023-12-15T16:51:22.763 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-15T16:51:22.763 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-15T16:51:22.763 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-15T16:51:22.763 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:22.763 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:51:22.763 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:22.763 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2023-12-15T16:51:22.763 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:22.764 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 42 M 2023-12-15T16:51:22.764 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 173 M 2023-12-15T16:51:22.764 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:51:22.798 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:51:22.798 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:51:22.915 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-12-15T16:51:22.935 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:51:22.957 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.2 MB/s | 217 kB 00:00 2023-12-15T16:51:22.998 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2023-12-15T16:51:23.040 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-15T16:51:23.051 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:23.199 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): libcephsqlite-19.0.0-157.g021ac167.el9.x 376 kB/s | 163 kB 00:00 2023-12-15T16:51:23.478 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:23.767 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:23.767 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:23.768 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:51:23.768 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:23.768 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:23.768 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:23.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-mirror 2023-12-15T16:51:24.546 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:36 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:24.693 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:24.693 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.0 M 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 3.0 M 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 12 M 2023-12-15T16:51:24.694 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:51:24.866 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): ceph-test-19.0.0-157.g021ac167.el9.x86_6 20 MB/s | 41 MB 00:02 2023-12-15T16:51:24.869 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:24.869 INFO:teuthology.orchestra.run.smithi192.stdout:Total 20 MB/s | 42 MB 00:02 2023-12-15T16:51:24.869 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:51:24.889 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:51:24.890 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:51:25.089 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-mirror-19.0.0-157.g021ac167.el9.x86_64.rpm 7.6 MB/s | 3.0 MB 00:00 2023-12-15T16:51:25.089 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:25.090 INFO:teuthology.orchestra.run.smithi069.stdout:Total 7.6 MB/s | 3.0 MB 00:00 2023-12-15T16:51:25.090 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:51:25.099 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:51:25.099 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:51:25.171 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:51:25.171 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:51:25.240 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:51:25.240 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:51:25.581 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:51:25.657 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:25.681 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:25.681 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:51:25.681 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-15T16:51:25.681 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-15T16:51:25.681 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-15T16:51:25.681 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:51:25.870 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-15T16:51:25.908 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-15T16:51:25.940 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-15T16:51:25.983 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-15T16:51:26.011 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 5/6 2023-12-15T16:51:26.857 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:26.857 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:26.857 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:51:26.858 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:26.858 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:26.858 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:27.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-nbd 2023-12-15T16:51:27.666 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:39 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:27.808 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:19.0.0-157.g021ac167.el9 ceph 170 k 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 170 k 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 490 k 2023-12-15T16:51:27.809 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:51:28.167 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-nbd-19.0.0-157.g021ac167.el9.x86_64.rpm 474 kB/s | 170 kB 00:00 2023-12-15T16:51:28.168 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:28.168 INFO:teuthology.orchestra.run.smithi069.stdout:Total 472 kB/s | 170 kB 00:00 2023-12-15T16:51:28.168 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:51:28.176 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:51:28.176 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:51:28.231 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:51:28.231 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:51:28.377 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:51:28.465 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:28.902 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:29.234 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:51:29.234 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:29.234 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:51:29.234 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:29.234 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:29.234 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:29.470 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base-debuginfo 2023-12-15T16:51:30.031 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:42 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:30.041 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 5/6 2023-12-15T16:51:30.151 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 6/6 2023-12-15T16:51:30.180 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 73 M 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 21 M 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2023-12-15T16:51:30.181 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:30.182 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 94 M 2023-12-15T16:51:30.182 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 508 M 2023-12-15T16:51:30.182 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:51:32.226 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 6/6 2023-12-15T16:51:32.226 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 1/6 2023-12-15T16:51:32.226 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 2/6 2023-12-15T16:51:32.226 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-15T16:51:32.227 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-15T16:51:32.227 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-15T16:51:32.262 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): ceph-debuginfo-19.0.0-157.g021ac167.el9. 9.9 MB/s | 21 MB 00:02 2023-12-15T16:51:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-15T16:51:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:32.627 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:51:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: jq-1.6-16.el9.x86_64 2023-12-15T16:51:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-15T16:51:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-15T16:51:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-15T16:51:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:32.627 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:51:32.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph 2023-12-15T16:51:33.462 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:40 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:51:33.619 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:51:33.621 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2023-12-15T16:51:33.621 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-12-15T16:51:33.621 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2023-12-15T16:51:33.621 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:19.0.0-157.g021ac167.el9 ceph 6.5 k 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:19.0.0-157.g021ac167.el9 ceph 2.1 M 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:19.0.0-157.g021ac167.el9 ceph 1.5 M 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 243 k 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:19.0.0-157.g021ac167.el9 ceph 4.4 M 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:19.0.0-157.g021ac167.el9 ceph 18 M 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-15T16:51:33.622 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-15T16:51:33.623 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout:Install 35 Packages 2023-12-15T16:51:33.624 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:33.625 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 30 M 2023-12-15T16:51:33.625 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 111 M 2023-12-15T16:51:33.625 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:51:35.435 INFO:teuthology.orchestra.run.smithi192.stdout:(1/35): ceph-19.0.0-157.g021ac167.el9.x86_64.rp 27 kB/s | 6.5 kB 00:00 2023-12-15T16:51:35.560 INFO:teuthology.orchestra.run.smithi192.stdout:(2/35): ceph-mgr-19.0.0-157.g021ac167.el9.x86_6 3.9 MB/s | 1.5 MB 00:00 2023-12-15T16:51:35.610 INFO:teuthology.orchestra.run.smithi192.stdout:(3/35): ceph-mds-19.0.0-157.g021ac167.el9.x86_6 4.9 MB/s | 2.1 MB 00:00 2023-12-15T16:51:35.677 INFO:teuthology.orchestra.run.smithi192.stdout:(4/35): ceph-mgr-modules-core-19.0.0-157.g021ac 3.6 MB/s | 243 kB 00:00 2023-12-15T16:51:35.786 INFO:teuthology.orchestra.run.smithi192.stdout:(5/35): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-12-15T16:51:35.828 INFO:teuthology.orchestra.run.smithi192.stdout:(6/35): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-12-15T16:51:35.861 INFO:teuthology.orchestra.run.smithi192.stdout:(7/35): python3-markupsafe-1.1.1-12.el9.x86_64. 954 kB/s | 32 kB 00:00 2023-12-15T16:51:35.895 INFO:teuthology.orchestra.run.smithi192.stdout:(8/35): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-15T16:51:35.928 INFO:teuthology.orchestra.run.smithi192.stdout:(9/35): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-15T16:51:35.970 INFO:teuthology.orchestra.run.smithi192.stdout:(10/35): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-12-15T16:51:36.037 INFO:teuthology.orchestra.run.smithi192.stdout:(11/35): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2023-12-15T16:51:36.063 INFO:teuthology.orchestra.run.smithi192.stdout:(12/35): python3-ply-3.11-14.el9.noarch.rpm 4.1 MB/s | 103 kB 00:00 2023-12-15T16:51:36.096 INFO:teuthology.orchestra.run.smithi192.stdout:(13/35): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-12-15T16:51:36.238 INFO:teuthology.orchestra.run.smithi192.stdout:(14/35): python3-requests-2.25.1-7.el9.noarch.r 795 kB/s | 113 kB 00:00 2023-12-15T16:51:36.272 INFO:teuthology.orchestra.run.smithi192.stdout:(15/35): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-12-15T16:51:36.556 INFO:teuthology.orchestra.run.smithi192.stdout:(16/35): ceph-osd-19.0.0-157.g021ac167.el9.x86_ 18 MB/s | 18 MB 00:00 2023-12-15T16:51:36.764 INFO:teuthology.orchestra.run.smithi192.stdout:(17/35): ceph-mon-19.0.0-157.g021ac167.el9.x86_ 3.3 MB/s | 4.4 MB 00:01 2023-12-15T16:51:36.939 INFO:teuthology.orchestra.run.smithi192.stdout:(18/35): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 114 kB/s | 43 kB 00:00 2023-12-15T16:51:36.981 INFO:teuthology.orchestra.run.smithi192.stdout:(19/35): luarocks-3.9.2-1.el9.noarch.rpm 213 kB/s | 151 kB 00:00 2023-12-15T16:51:37.040 INFO:teuthology.orchestra.run.smithi192.stdout:(20/35): python3-cheroot-8.6.0-4.el9.noarch.rpm 624 kB/s | 172 kB 00:00 2023-12-15T16:51:37.157 INFO:teuthology.orchestra.run.smithi192.stdout:(21/35): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2023-12-15T16:51:37.182 INFO:teuthology.orchestra.run.smithi192.stdout:(22/35): python3-jaraco-8.2.1-3.el9.noarch.rpm 53 kB/s | 11 kB 00:00 2023-12-15T16:51:37.207 INFO:teuthology.orchestra.run.smithi192.stdout:(23/35): python3-jaraco-classes-3.2.1-5.el9.noa 106 kB/s | 18 kB 00:00 2023-12-15T16:51:37.274 INFO:teuthology.orchestra.run.smithi192.stdout:(24/35): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2023-12-15T16:51:37.308 INFO:teuthology.orchestra.run.smithi192.stdout:(25/35): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2023-12-15T16:51:37.333 INFO:teuthology.orchestra.run.smithi192.stdout:(26/35): python3-jaraco-text-3.2.0-6.el9.noarch 157 kB/s | 20 kB 00:00 2023-12-15T16:51:37.400 INFO:teuthology.orchestra.run.smithi192.stdout:(27/35): python3-logutils-0.3.5-21.el9.noarch.r 369 kB/s | 46 kB 00:00 2023-12-15T16:51:37.425 INFO:teuthology.orchestra.run.smithi192.stdout:(28/35): python3-more-itertools-8.12.0-2.el9.no 673 kB/s | 79 kB 00:00 2023-12-15T16:51:37.525 INFO:teuthology.orchestra.run.smithi192.stdout:(29/35): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2023-12-15T16:51:37.551 INFO:teuthology.orchestra.run.smithi192.stdout:(30/35): python3-portend-3.1.0-2.el9.noarch.rpm 109 kB/s | 16 kB 00:00 2023-12-15T16:51:37.567 INFO:teuthology.orchestra.run.smithi192.stdout:(31/35): python3-pyOpenSSL-21.0.0-1.el9.noarch. 634 kB/s | 90 kB 00:00 2023-12-15T16:51:37.651 INFO:teuthology.orchestra.run.smithi192.stdout:(32/35): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2023-12-15T16:51:37.726 INFO:teuthology.orchestra.run.smithi192.stdout:(33/35): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2023-12-15T16:51:37.768 INFO:teuthology.orchestra.run.smithi192.stdout:(34/35): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.1 MB/s | 427 kB 00:00 2023-12-15T16:51:37.794 INFO:teuthology.orchestra.run.smithi192.stdout:(35/35): python3-zc-lockfile-2.0-10.el9.noarch. 141 kB/s | 20 kB 00:00 2023-12-15T16:51:37.798 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:37.798 INFO:teuthology.orchestra.run.smithi192.stdout:Total 7.2 MB/s | 30 MB 00:04 2023-12-15T16:51:37.799 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:51:37.840 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:51:37.840 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:51:38.244 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:51:38.244 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:51:39.565 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:51:39.621 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/35 2023-12-15T16:51:39.653 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/35 2023-12-15T16:51:39.686 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/35 2023-12-15T16:51:39.777 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/35 2023-12-15T16:51:39.856 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/35 2023-12-15T16:51:39.897 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/35 2023-12-15T16:51:39.968 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/35 2023-12-15T16:51:40.022 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/35 2023-12-15T16:51:40.070 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/35 2023-12-15T16:51:40.118 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/35 2023-12-15T16:51:40.174 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/35 2023-12-15T16:51:40.357 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-15T16:51:40.399 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-15T16:51:40.444 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/35 2023-12-15T16:51:40.480 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/35 2023-12-15T16:51:40.510 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/35 2023-12-15T16:51:40.586 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/35 2023-12-15T16:51:40.621 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/35 2023-12-15T16:51:40.654 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/35 2023-12-15T16:51:40.695 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/35 2023-12-15T16:51:40.729 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/35 2023-12-15T16:51:40.764 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/35 2023-12-15T16:51:40.862 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/35 2023-12-15T16:51:40.895 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/35 2023-12-15T16:51:40.967 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/35 2023-12-15T16:51:41.104 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/35 2023-12-15T16:51:41.229 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/35 2023-12-15T16:51:41.382 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9.n 28/35 2023-12-15T16:51:41.408 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 29/35 2023-12-15T16:51:41.432 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 29/35 2023-12-15T16:51:41.432 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:51:41.433 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-15T16:51:41.433 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-15T16:51:41.433 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-15T16:51:41.433 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-5.4.4-4.el9.x86_64 30/35 2023-12-15T16:51:41.569 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-base-debuginfo-19.0.0-157.g021ac167 6.4 MB/s | 73 MB 00:11 2023-12-15T16:51:41.571 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:41.571 INFO:teuthology.orchestra.run.smithi069.stdout:Total 8.2 MB/s | 94 MB 00:11 2023-12-15T16:51:41.572 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:51:41.576 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:51:41.576 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:51:42.176 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:51:42.177 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:51:43.095 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : luarocks-3.9.2-1.el9.noarch 31/35 2023-12-15T16:51:43.128 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 32/35 2023-12-15T16:51:43.157 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 32/35 2023-12-15T16:51:43.157 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:51:43.157 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-15T16:51:43.157 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-15T16:51:43.157 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-15T16:51:43.157 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:43.604 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 33/35 2023-12-15T16:51:43.628 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 33/35 2023-12-15T16:51:43.628 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:51:43.628 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-15T16:51:43.628 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-15T16:51:43.628 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-15T16:51:43.628 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:43.830 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 34/35 2023-12-15T16:51:43.853 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 34/35 2023-12-15T16:51:43.853 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:51:43.853 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-15T16:51:43.853 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-15T16:51:43.853 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-15T16:51:43.853 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:19.0.0-157.g021ac167.el9.x86_64 35/35 2023-12-15T16:51:45.097 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:51:45.608 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-2:19.0.0-157.g021ac167.el9.x86_64 35/35 2023-12-15T16:51:45.608 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:19.0.0-157.g021ac167.el9.x86_64 1/35 2023-12-15T16:51:45.608 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 2/35 2023-12-15T16:51:45.608 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 3/35 2023-12-15T16:51:45.608 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 4/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 5/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9.n 6/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/35 2023-12-15T16:51:45.609 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/35 2023-12-15T16:51:45.610 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 18/35 2023-12-15T16:51:45.610 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 19/35 2023-12-15T16:51:45.610 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 20/35 2023-12-15T16:51:45.610 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 21/35 2023-12-15T16:51:45.610 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 22/35 2023-12-15T16:51:45.610 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 23/35 2023-12-15T16:51:45.610 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 24/35 2023-12-15T16:51:45.610 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 25/35 2023-12-15T16:51:45.610 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 26/35 2023-12-15T16:51:45.610 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 27/35 2023-12-15T16:51:45.611 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 28/35 2023-12-15T16:51:45.611 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 29/35 2023-12-15T16:51:45.611 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 30/35 2023-12-15T16:51:45.611 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 31/35 2023-12-15T16:51:45.611 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 32/35 2023-12-15T16:51:45.611 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 33/35 2023-12-15T16:51:45.611 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 34/35 2023-12-15T16:51:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 35/35 2023-12-15T16:51:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:46.567 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:51:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:51:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: lua-5.4.4-4.el9.x86_64 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-15T16:51:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-15T16:51:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-15T16:51:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-15T16:51:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-15T16:51:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply-3.11-14.el9.noarch 2023-12-15T16:51:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-15T16:51:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-15T16:51:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-15T16:51:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-15T16:51:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-15T16:51:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-15T16:51:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-15T16:51:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-15T16:51:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-15T16:51:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-15T16:51:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-15T16:51:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:46.570 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:51:46.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base 2023-12-15T16:51:47.514 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:54 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:51:47.607 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:51:47.656 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:51:47.657 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-12-15T16:51:47.657 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:51:47.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install cephadm 2023-12-15T16:51:48.281 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:55 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:51:48.422 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:51:48.423 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:48.423 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-12-15T16:51:48.424 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:48.424 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:51:48.424 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 803 k 2023-12-15T16:51:48.424 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:48.424 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:51:48.424 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:48.424 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:51:48.424 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:48.424 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 803 k 2023-12-15T16:51:48.424 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 818 k 2023-12-15T16:51:48.424 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:51:48.704 INFO:teuthology.orchestra.run.smithi192.stdout:cephadm-19.0.0-157.g021ac167.el9.noarch.rpm 2.8 MB/s | 803 kB 00:00 2023-12-15T16:51:48.705 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:48.705 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.8 MB/s | 803 kB 00:00 2023-12-15T16:51:48.705 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:51:48.710 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:51:48.711 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:51:48.716 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:51:48.717 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:51:48.792 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:51:49.230 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:51:49.318 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:51:49.702 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:51:50.028 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:51:50.028 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:50.028 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:51:50.028 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:51:50.028 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:50.028 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:51:50.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache 2023-12-15T16:51:50.800 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:57 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:51:50.949 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:51:50.949 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-157.g021ac167.el9 ceph 141 k 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 141 k 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 395 k 2023-12-15T16:51:50.950 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:51:51.262 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-19.0.0-157.g021ac16 456 kB/s | 141 kB 00:00 2023-12-15T16:51:51.263 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:51.263 INFO:teuthology.orchestra.run.smithi192.stdout:Total 454 kB/s | 141 kB 00:00 2023-12-15T16:51:51.263 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:51:51.268 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:51:51.268 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:51:51.320 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:51:51.320 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:51:51.461 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:51:51.528 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-157.g021ac167.e 1/1 2023-12-15T16:51:51.552 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-157.g021ac167.e 1/1 2023-12-15T16:51:51.552 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:51:51.552 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-15T16:51:51.552 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:52.529 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-157.g021ac167.e 1/1 2023-12-15T16:51:52.529 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:52.529 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:51:52.529 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:52.529 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:52.529 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:51:52.660 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/2 2023-12-15T16:51:52.735 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 2/2 2023-12-15T16:51:52.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr 2023-12-15T16:51:53.327 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:00 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:51:53.421 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:51:53.471 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:51:53.472 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-12-15T16:51:53.472 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:51:53.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-dashboard 2023-12-15T16:51:54.093 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:01 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:51:54.238 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:51:54.238 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 1.8 M 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 28 k 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 15 k 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:51:54.239 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:51:54.240 INFO:teuthology.orchestra.run.smithi192.stdout:Install 7 Packages 2023-12-15T16:51:54.240 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:54.240 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 2.1 M 2023-12-15T16:51:54.240 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 8.7 M 2023-12-15T16:51:54.240 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:51:54.580 INFO:teuthology.orchestra.run.smithi192.stdout:(1/7): ceph-grafana-dashboards-19.0.0-157.g021a 108 kB/s | 28 kB 00:00 2023-12-15T16:51:54.605 INFO:teuthology.orchestra.run.smithi192.stdout:(2/7): ceph-prometheus-alerts-19.0.0-157.g021ac 53 kB/s | 15 kB 00:00 2023-12-15T16:51:54.847 INFO:teuthology.orchestra.run.smithi192.stdout:(3/7): ceph-mgr-dashboard-19.0.0-157.g021ac167. 3.3 MB/s | 1.8 MB 00:00 2023-12-15T16:51:54.906 INFO:teuthology.orchestra.run.smithi192.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 28 kB/s | 9.0 kB 00:00 2023-12-15T16:51:54.989 INFO:teuthology.orchestra.run.smithi192.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 107 kB/s | 41 kB 00:00 2023-12-15T16:51:55.214 INFO:teuthology.orchestra.run.smithi192.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 84 kB/s | 31 kB 00:00 2023-12-15T16:51:55.281 INFO:teuthology.orchestra.run.smithi192.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 502 kB/s | 188 kB 00:00 2023-12-15T16:51:55.285 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:51:55.286 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.0 MB/s | 2.1 MB 00:01 2023-12-15T16:51:55.286 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:51:55.293 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:51:55.293 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:51:55.357 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:51:55.357 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:51:55.652 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:51:55.708 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-15T16:51:55.741 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-15T16:51:55.795 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-15T16:51:55.828 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-15T16:51:55.860 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el9.no 5/7 2023-12-15T16:51:56.098 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-157.g021ac167.el9.n 6/7 2023-12-15T16:51:56.123 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 2/2 2023-12-15T16:51:56.124 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/2 2023-12-15T16:51:56.129 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.noarch 7/7 2023-12-15T16:51:56.401 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2/2 2023-12-15T16:51:56.401 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:56.401 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:51:56.401 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:56.401 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:51:56.401 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:56.401 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:51:56.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-common-debuginfo 2023-12-15T16:51:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.noarch 7/7 2023-12-15T16:51:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-157.g021ac167.el9.n 1/7 2023-12-15T16:51:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.noarch 2/7 2023-12-15T16:51:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el9.no 3/7 2023-12-15T16:51:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-15T16:51:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-15T16:51:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-15T16:51:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-15T16:51:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:56.993 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:51:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:51:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:51:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:51:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-15T16:51:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-15T16:51:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-15T16:51:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-15T16:51:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:56.993 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:51:57.200 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:02:09 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:51:57.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-15T16:51:57.349 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:51:57.350 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:57.350 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:51:57.350 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:57.350 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:51:57.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 405 M 2023-12-15T16:51:57.350 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:57.350 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:51:57.350 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:51:57.350 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:51:57.350 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:51:57.351 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 405 M 2023-12-15T16:51:57.351 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.6 G 2023-12-15T16:51:57.351 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:51:57.781 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:04 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:51:57.925 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout:=========================================================================================== 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout:=========================================================================================== 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 7.4 M 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-15T16:51:57.926 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-15T16:51:57.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-15T16:51:57.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-15T16:51:57.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-15T16:51:57.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-15T16:51:57.927 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:57.927 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:51:57.927 INFO:teuthology.orchestra.run.smithi192.stdout:=========================================================================================== 2023-12-15T16:51:57.927 INFO:teuthology.orchestra.run.smithi192.stdout:Install 12 Packages 2023-12-15T16:51:57.927 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:51:57.928 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 37 M 2023-12-15T16:51:57.928 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 209 M 2023-12-15T16:51:57.928 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:51:58.173 INFO:teuthology.orchestra.run.smithi192.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 257 kB/s | 32 kB 00:00 2023-12-15T16:51:58.207 INFO:teuthology.orchestra.run.smithi192.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 524 kB/s | 17 kB 00:00 2023-12-15T16:51:58.265 INFO:teuthology.orchestra.run.smithi192.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2023-12-15T16:51:58.291 INFO:teuthology.orchestra.run.smithi192.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 419 kB/s | 34 kB 00:00 2023-12-15T16:51:58.416 INFO:teuthology.orchestra.run.smithi192.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 32 MB/s | 4.7 MB 00:00 2023-12-15T16:51:58.483 INFO:teuthology.orchestra.run.smithi192.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2023-12-15T16:51:58.508 INFO:teuthology.orchestra.run.smithi192.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-12-15T16:51:58.567 INFO:teuthology.orchestra.run.smithi192.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 14 MB/s | 803 kB 00:00 2023-12-15T16:51:58.717 INFO:teuthology.orchestra.run.smithi192.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 67 MB/s | 16 MB 00:00 2023-12-15T16:51:58.742 INFO:teuthology.orchestra.run.smithi192.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-12-15T16:51:59.343 INFO:teuthology.orchestra.run.smithi192.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 393 kB/s | 245 kB 00:00 2023-12-15T16:52:09.167 INFO:teuthology.orchestra.run.smithi192.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 682 kB/s | 7.4 MB 00:11 2023-12-15T16:52:09.171 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:09.171 INFO:teuthology.orchestra.run.smithi192.stdout:Total 3.3 MB/s | 37 MB 00:11 2023-12-15T16:52:09.185 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:52:09.211 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:52:09.211 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:52:09.597 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:52:09.597 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:52:10.306 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:52:10.381 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-15T16:52:10.413 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-15T16:52:10.505 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-15T16:52:10.541 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-15T16:52:10.719 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-15T16:52:10.756 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-15T16:52:10.847 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-15T16:52:11.495 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-15T16:52:11.602 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-15T16:52:12.840 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-15T16:52:13.861 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-15T16:52:13.889 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-157.g021ac1 12/12 2023-12-15T16:52:16.329 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-157.g021ac1 12/12 2023-12-15T16:52:16.329 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-157.g021ac1 1/12 2023-12-15T16:52:16.329 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-15T16:52:16.329 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-15T16:52:16.329 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-15T16:52:16.329 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-15T16:52:16.329 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-15T16:52:16.329 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-15T16:52:16.329 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-15T16:52:16.330 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-15T16:52:16.330 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-15T16:52:16.330 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-15T16:52:16.781 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-15T16:52:16.781 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:16.781 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:52:16.781 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:52:16.781 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-15T16:52:16.781 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-15T16:52:16.781 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-15T16:52:16.781 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-15T16:52:16.781 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-15T16:52:16.781 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-15T16:52:16.782 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-15T16:52:16.782 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-15T16:52:16.782 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-15T16:52:16.782 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-15T16:52:16.782 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-15T16:52:16.782 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:16.782 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:17.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-rook 2023-12-15T16:52:17.736 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:24 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:17.879 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:17.880 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2023-12-15T16:52:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-12-15T16:52:17.880 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 50 k 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-15T16:52:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-15T16:52:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-15T16:52:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-15T16:52:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-15T16:52:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:17.882 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:52:17.882 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2023-12-15T16:52:17.882 INFO:teuthology.orchestra.run.smithi192.stdout:Install 13 Packages 2023-12-15T16:52:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:17.883 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 2.0 M 2023-12-15T16:52:17.883 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 26 M 2023-12-15T16:52:17.883 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:52:18.065 INFO:teuthology.orchestra.run.smithi192.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 137 kB/s | 18 kB 00:00 2023-12-15T16:52:18.090 INFO:teuthology.orchestra.run.smithi192.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 155 kB/s | 24 kB 00:00 2023-12-15T16:52:18.132 INFO:teuthology.orchestra.run.smithi192.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-12-15T16:52:18.157 INFO:teuthology.orchestra.run.smithi192.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-15T16:52:18.183 INFO:teuthology.orchestra.run.smithi192.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-15T16:52:18.208 INFO:teuthology.orchestra.run.smithi192.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2023-12-15T16:52:18.242 INFO:teuthology.orchestra.run.smithi192.stdout:(7/13): ceph-mgr-rook-19.0.0-157.g021ac167.el9. 163 kB/s | 50 kB 00:00 2023-12-15T16:52:18.583 INFO:teuthology.orchestra.run.smithi192.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 80 kB/s | 32 kB 00:00 2023-12-15T16:52:18.609 INFO:teuthology.orchestra.run.smithi192.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 37 kB/s | 15 kB 00:00 2023-12-15T16:52:18.843 INFO:teuthology.orchestra.run.smithi192.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2023-12-15T16:52:18.868 INFO:teuthology.orchestra.run.smithi192.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 315 kB/s | 197 kB 00:00 2023-12-15T16:52:19.043 INFO:teuthology.orchestra.run.smithi192.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 448 kB/s | 90 kB 00:00 2023-12-15T16:52:19.077 INFO:teuthology.orchestra.run.smithi192.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2023-12-15T16:52:19.081 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:19.081 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2023-12-15T16:52:19.081 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:52:19.094 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:52:19.094 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:52:19.190 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:52:19.192 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:52:19.712 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:52:19.871 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-15T16:52:19.936 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-15T16:52:19.990 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-15T16:52:20.020 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-15T16:52:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-15T16:52:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-15T16:52:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-15T16:52:20.313 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-15T16:52:21.022 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-15T16:52:21.077 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-15T16:52:21.110 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-15T16:52:21.174 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-15T16:52:21.205 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 13/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 13/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 1/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-15T16:52:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-15T16:52:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-15T16:52:22.358 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-15T16:52:22.358 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:22.358 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:52:22.358 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:52:22.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-15T16:52:22.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-15T16:52:22.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-15T16:52:22.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-15T16:52:22.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-15T16:52:22.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-15T16:52:22.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-15T16:52:22.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-15T16:52:22.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-15T16:52:22.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-15T16:52:22.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-15T16:52:22.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-15T16:52:22.359 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:22.359 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:22.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-cephadm 2023-12-15T16:52:23.186 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:30 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:23.330 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:23.331 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2023-12-15T16:52:23.331 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-12-15T16:52:23.331 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2023-12-15T16:52:23.331 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:52:23.331 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 133 k 2023-12-15T16:52:23.331 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-12-15T16:52:23.331 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-15T16:52:23.331 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-15T16:52:23.331 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-15T16:52:23.331 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-15T16:52:23.331 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-15T16:52:23.332 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:23.332 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:52:23.332 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2023-12-15T16:52:23.332 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2023-12-15T16:52:23.332 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:23.332 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 6.8 M 2023-12-15T16:52:23.332 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 33 M 2023-12-15T16:52:23.332 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:52:23.653 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-12-15T16:52:23.770 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-12-15T16:52:23.878 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): ceph-mgr-cephadm-19.0.0-157.g021ac167.el 335 kB/s | 133 kB 00:00 2023-12-15T16:52:24.004 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 274 kB/s | 58 kB 00:00 2023-12-15T16:52:24.062 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2023-12-15T16:52:24.087 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 244 kB/s | 51 kB 00:00 2023-12-15T16:52:24.091 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:24.091 INFO:teuthology.orchestra.run.smithi192.stdout:Total 8.9 MB/s | 6.8 MB 00:00 2023-12-15T16:52:24.091 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:52:24.100 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:52:24.100 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:52:24.188 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:52:24.189 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:52:24.483 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:52:24.649 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-15T16:52:24.705 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-15T16:52:25.214 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-15T16:52:25.290 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-15T16:52:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-15T16:52:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noarch 6/6 2023-12-15T16:52:26.194 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noarch 6/6 2023-12-15T16:52:26.194 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noarch 1/6 2023-12-15T16:52:26.194 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-15T16:52:26.194 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-15T16:52:26.194 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-15T16:52:26.194 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-15T16:52:26.508 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-15T16:52:26.508 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:26.508 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:52:26.508 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:52:26.508 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-15T16:52:26.508 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-15T16:52:26.508 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-15T16:52:26.508 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-15T16:52:26.508 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-15T16:52:26.508 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:26.508 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:26.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse 2023-12-15T16:52:27.339 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:34 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:27.484 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:19.0.0-157.g021ac167.el9 ceph 859 k 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2023-12-15T16:52:27.485 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:27.486 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 938 k 2023-12-15T16:52:27.486 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.7 M 2023-12-15T16:52:27.486 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:52:27.614 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 617 kB/s | 79 kB 00:00 2023-12-15T16:52:27.814 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-fuse-19.0.0-157.g021ac167.el9.x86_6 2.6 MB/s | 859 kB 00:00 2023-12-15T16:52:27.816 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:27.816 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.8 MB/s | 938 kB 00:00 2023-12-15T16:52:27.816 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:52:27.837 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:52:27.837 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:52:27.912 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:52:27.912 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:52:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:52:28.247 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-15T16:52:28.322 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 2/2 2023-12-15T16:52:28.963 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 2/2 2023-12-15T16:52:28.963 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 1/2 2023-12-15T16:52:29.286 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-15T16:52:29.286 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:29.286 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:52:29.286 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-15T16:52:29.286 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:29.286 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:29.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-volume 2023-12-15T16:52:30.111 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:37 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:30.259 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:30.260 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:30.261 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-12-15T16:52:30.261 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:30.261 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:52:30.261 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 262 k 2023-12-15T16:52:30.261 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:30.261 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:52:30.261 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:30.262 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:52:30.262 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:30.262 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 262 k 2023-12-15T16:52:30.262 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.2 M 2023-12-15T16:52:30.262 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:52:30.502 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-volume-19.0.0-157.g021ac167.el9.noarch.rpm 1.1 MB/s | 262 kB 00:00 2023-12-15T16:52:30.502 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:30.503 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-12-15T16:52:30.503 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:52:30.511 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:52:30.511 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:52:30.550 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:52:30.550 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:52:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:52:30.837 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:52:30.853 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:52:30.853 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:52:30.853 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-15T16:52:30.853 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:31.840 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 1/1 2023-12-15T16:52:31.840 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:31.840 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:52:31.840 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 2023-12-15T16:52:31.841 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:31.841 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:32.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel 2023-12-15T16:52:32.738 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:39 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:32.883 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:32.884 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:32.884 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:52:32.884 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:32.884 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:52:32.884 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:19.0.0-157.g021ac167.el9 ceph 126 k 2023-12-15T16:52:32.884 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:32.884 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:52:32.884 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:32.885 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:52:32.885 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:32.885 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 126 k 2023-12-15T16:52:32.885 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 456 k 2023-12-15T16:52:32.885 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:52:33.223 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-19.0.0-157.g021ac167.el9.x86_64. 374 kB/s | 126 kB 00:00 2023-12-15T16:52:33.224 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:33.224 INFO:teuthology.orchestra.run.smithi192.stdout:Total 373 kB/s | 126 kB 00:00 2023-12-15T16:52:33.224 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:52:33.229 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:52:33.229 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:52:33.285 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:52:33.285 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:52:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:52:33.529 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:33.989 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:34.286 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:34.286 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:34.286 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:52:34.286 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:52:34.286 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:34.286 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:34.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2 2023-12-15T16:52:35.114 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:42 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:35.211 INFO:teuthology.orchestra.run.smithi192.stdout:Package libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:52:35.262 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:35.262 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-12-15T16:52:35.263 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:35.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs-devel 2023-12-15T16:52:35.892 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:42 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:36.035 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:36.036 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:36.036 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:52:36.036 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:36.036 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:52:36.036 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:19.0.0-157.g021ac167.el9 ceph 31 k 2023-12-15T16:52:36.036 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:36.036 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:52:36.036 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:36.036 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:52:36.036 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:36.037 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 31 k 2023-12-15T16:52:36.037 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 139 k 2023-12-15T16:52:36.037 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:52:36.277 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs-devel-19.0.0-157.g021ac167.el9.x86_64 130 kB/s | 31 kB 00:00 2023-12-15T16:52:36.278 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:36.278 INFO:teuthology.orchestra.run.smithi192.stdout:Total 129 kB/s | 31 kB 00:00 2023-12-15T16:52:36.278 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:52:36.281 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:52:36.281 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:52:36.306 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:52:36.306 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:52:36.395 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:52:36.498 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:37.161 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:52:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:52:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:37.161 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:37.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2 2023-12-15T16:52:37.394 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-common-debuginfo-19.0.0-157.g021ac167.el9. 10 MB/s | 405 MB 00:40 2023-12-15T16:52:37.395 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:37.396 INFO:teuthology.orchestra.run.smithi069.stdout:Total 10 MB/s | 405 MB 00:40 2023-12-15T16:52:37.396 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:52:37.400 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:52:37.400 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:52:37.922 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:44 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:38.015 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:52:38.067 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:38.068 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-12-15T16:52:38.069 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:38.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1 2023-12-15T16:52:38.711 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:45 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:38.806 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:52:38.856 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:38.857 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-12-15T16:52:38.857 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:38.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados 2023-12-15T16:52:39.485 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:46 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:39.579 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:52:39.632 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:39.633 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-12-15T16:52:39.633 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:39.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw 2023-12-15T16:52:39.799 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:52:39.799 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:52:40.264 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:47 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:40.358 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:52:40.408 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:40.409 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-12-15T16:52:40.409 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:40.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs 2023-12-15T16:52:41.045 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:48 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:41.139 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:52:41.188 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:41.189 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-12-15T16:52:41.190 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:41.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd 2023-12-15T16:52:41.818 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:48 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:41.910 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:52:41.960 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:41.961 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-12-15T16:52:41.961 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:42.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse 2023-12-15T16:52:42.588 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:49 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:42.733 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:42.734 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:42.734 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2023-12-15T16:52:42.734 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:42.734 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:52:42.734 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:19.0.0-157.g021ac167.el9 ceph 85 k 2023-12-15T16:52:42.734 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:42.735 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:52:42.735 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:42.735 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:52:42.735 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:42.735 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 85 k 2023-12-15T16:52:42.735 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 231 k 2023-12-15T16:52:42.735 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:52:43.046 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-19.0.0-157.g021ac167.el9.x86_64.rpm 273 kB/s | 85 kB 00:00 2023-12-15T16:52:43.047 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:43.047 INFO:teuthology.orchestra.run.smithi192.stdout:Total 272 kB/s | 85 kB 00:00 2023-12-15T16:52:43.047 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:52:43.054 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:52:43.054 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:52:43.108 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:52:43.108 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:52:43.246 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:52:43.343 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:43.750 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:44.071 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:44.071 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:44.071 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:52:44.071 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:52:44.071 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:44.072 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:44.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror 2023-12-15T16:52:44.877 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:51 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:45.031 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:45.032 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:45.032 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:52:45.032 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:45.032 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:52:45.032 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.0 M 2023-12-15T16:52:45.032 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:45.032 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:52:45.032 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:45.032 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:52:45.033 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:45.033 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.0 M 2023-12-15T16:52:45.033 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 12 M 2023-12-15T16:52:45.033 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:52:45.574 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-19.0.0-157.g021ac167.el9.x86_64.rpm 5.6 MB/s | 3.0 MB 00:00 2023-12-15T16:52:45.575 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:45.576 INFO:teuthology.orchestra.run.smithi192.stdout:Total 5.5 MB/s | 3.0 MB 00:00 2023-12-15T16:52:45.576 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:52:45.585 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:52:45.585 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:52:45.661 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:52:45.661 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:52:46.076 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:52:46.126 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:46.151 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:46.151 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T16:52:46.151 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-15T16:52:46.151 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-15T16:52:46.152 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-15T16:52:46.152 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:47.375 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:52:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:52:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:47.375 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:47.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd 2023-12-15T16:52:48.209 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:55 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:48.357 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:48.358 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:48.358 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2023-12-15T16:52:48.358 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:48.358 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:52:48.358 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:19.0.0-157.g021ac167.el9 ceph 170 k 2023-12-15T16:52:48.358 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:48.358 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:52:48.358 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:48.358 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:52:48.358 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:48.359 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 170 k 2023-12-15T16:52:48.359 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 490 k 2023-12-15T16:52:48.359 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:52:48.711 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-19.0.0-157.g021ac167.el9.x86_64.rpm 482 kB/s | 170 kB 00:00 2023-12-15T16:52:48.712 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:48.712 INFO:teuthology.orchestra.run.smithi192.stdout:Total 480 kB/s | 170 kB 00:00 2023-12-15T16:52:48.712 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:52:48.720 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:52:48.720 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:52:48.775 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:52:48.776 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:52:48.920 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:52:49.009 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:49.454 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:49.774 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:52:49.774 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:49.774 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:52:49.774 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:52:49.774 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:49.774 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:52:50.021 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base-debuginfo 2023-12-15T16:52:50.592 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:57 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:52:50.741 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:52:50.742 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:50.742 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:52:50.742 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:50.742 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:52:50.742 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 73 M 2023-12-15T16:52:50.742 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-12-15T16:52:50.742 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 21 M 2023-12-15T16:52:50.742 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:50.742 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:52:50.743 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:52:50.743 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2023-12-15T16:52:50.743 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:52:50.743 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 94 M 2023-12-15T16:52:50.743 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 508 M 2023-12-15T16:52:50.743 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:52:54.280 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): ceph-base-debuginfo-19.0.0-157.g021ac167 21 MB/s | 73 MB 00:03 2023-12-15T16:52:57.780 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-debuginfo-19.0.0-157.g021ac167.el9. 2.9 MB/s | 21 MB 00:07 2023-12-15T16:52:57.782 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:52:57.783 INFO:teuthology.orchestra.run.smithi192.stdout:Total 13 MB/s | 94 MB 00:07 2023-12-15T16:52:57.783 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:52:57.788 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:52:57.788 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:52:58.394 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:52:58.394 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:53:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:53:08.897 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/2 2023-12-15T16:53:09.064 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 2/2 2023-12-15T16:53:12.624 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 2/2 2023-12-15T16:53:12.625 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/2 2023-12-15T16:53:12.907 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2/2 2023-12-15T16:53:12.907 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:53:12.907 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:53:12.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:53:12.908 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:53:12.908 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:53:12.908 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:53:13.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-common-debuginfo 2023-12-15T16:53:13.658 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:20 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:53:13.803 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:53:13.803 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:53:13.803 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:53:13.804 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:53:13.804 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:53:13.804 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 405 M 2023-12-15T16:53:13.804 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:53:13.804 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:53:13.804 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:53:13.804 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:53:13.804 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:53:13.804 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 405 M 2023-12-15T16:53:13.804 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.6 G 2023-12-15T16:53:13.804 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:53:20.726 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:53:20.821 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:53:28.364 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:53:28.636 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:53:28.636 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:53:28.636 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:53:28.636 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:53:28.636 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:53:28.636 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:53:28.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-15T16:53:29.450 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:03:41 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:53:29.599 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 1.4 M 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.4 M 2023-12-15T16:53:29.600 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 6.2 M 2023-12-15T16:53:29.601 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:53:30.326 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-15 1.9 MB/s | 1.4 MB 00:00 2023-12-15T16:53:30.327 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:53:30.327 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.9 MB/s | 1.4 MB 00:00 2023-12-15T16:53:30.328 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:53:30.333 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:53:30.333 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:53:30.394 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:53:30.394 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:53:30.652 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:53:30.757 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-157.g 1/1 2023-12-15T16:53:31.170 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-157.g 1/1 2023-12-15T16:53:31.437 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-157.g 1/1 2023-12-15T16:53:31.437 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:53:31.437 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:53:31.437 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:53:31.437 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:53:31.438 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:53:31.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-15T16:53:31.749 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-common-debuginfo-19.0.0-157.g021ac167.el9. 23 MB/s | 405 MB 00:17 2023-12-15T16:53:31.750 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:53:31.750 INFO:teuthology.orchestra.run.smithi192.stdout:Total 23 MB/s | 405 MB 00:17 2023-12-15T16:53:31.751 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:53:31.755 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:53:31.755 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:53:32.165 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:03:44 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:53:32.310 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:53:32.310 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:53:32.310 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:53:32.311 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:53:32.311 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:53:32.311 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 173 M 2023-12-15T16:53:32.311 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:53:32.311 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:53:32.311 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:53:32.311 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:53:32.311 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:53:32.312 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 173 M 2023-12-15T16:53:32.312 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 731 M 2023-12-15T16:53:32.312 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:53:34.153 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:53:34.154 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:53:41.011 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-radosgw-debuginfo-19.0.0-157.g021ac167.el9 20 MB/s | 173 MB 00:08 2023-12-15T16:53:41.012 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:53:41.013 INFO:teuthology.orchestra.run.smithi069.stdout:Total 20 MB/s | 173 MB 00:08 2023-12-15T16:53:41.013 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:53:41.017 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:53:41.017 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:53:42.057 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:53:42.057 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:53:59.904 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:54:00.007 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-157.g021ac167.el9.x8 1/1 2023-12-15T16:54:04.422 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-157.g021ac167.el9.x8 1/1 2023-12-15T16:54:04.699 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-157.g021ac167.el9.x8 1/1 2023-12-15T16:54:04.699 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:54:04.700 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:54:04.700 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:54:04.700 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:54:04.700 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:54:04.906 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-test-debuginfo 2023-12-15T16:54:05.466 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:17 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:54:05.608 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:54:05.609 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:54:05.609 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:54:05.609 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:54:05.609 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:54:05.609 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 623 M 2023-12-15T16:54:05.609 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:54:05.609 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:54:05.609 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:54:05.609 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:54:05.609 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:54:05.609 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 623 M 2023-12-15T16:54:05.610 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.8 G 2023-12-15T16:54:05.610 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:54:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:54:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:54:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:54:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:54:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:54:24.047 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:54:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:54:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:54:24.047 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:54:24.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-15T16:54:24.854 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:31 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:54:25.009 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:54:25.010 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2023-12-15T16:54:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:54:25.010 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2023-12-15T16:54:25.010 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:54:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 1.4 M 2023-12-15T16:54:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:54:25.011 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:54:25.011 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2023-12-15T16:54:25.011 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:54:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:54:25.011 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.4 M 2023-12-15T16:54:25.011 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 6.2 M 2023-12-15T16:54:25.011 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:54:25.332 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-15 4.3 MB/s | 1.4 MB 00:00 2023-12-15T16:54:25.332 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:54:25.332 INFO:teuthology.orchestra.run.smithi192.stdout:Total 4.3 MB/s | 1.4 MB 00:00 2023-12-15T16:54:25.333 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:54:25.336 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:54:25.336 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:54:25.379 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:54:25.379 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:54:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:54:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-157.g 1/1 2023-12-15T16:54:26.176 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-157.g 1/1 2023-12-15T16:54:26.450 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-157.g 1/1 2023-12-15T16:54:26.450 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:54:26.450 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:54:26.450 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:54:26.451 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:54:26.451 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:54:26.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-15T16:54:27.174 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:34 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:54:27.320 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:54:27.320 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:54:27.320 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:54:27.320 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:54:27.320 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:54:27.320 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 173 M 2023-12-15T16:54:27.320 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:54:27.320 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:54:27.321 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:54:27.321 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:54:27.321 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:54:27.321 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 173 M 2023-12-15T16:54:27.321 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 731 M 2023-12-15T16:54:27.321 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:54:39.558 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-radosgw-debuginfo-19.0.0-157.g021ac167.el9 14 MB/s | 173 MB 00:12 2023-12-15T16:54:39.559 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:54:39.560 INFO:teuthology.orchestra.run.smithi192.stdout:Total 14 MB/s | 173 MB 00:12 2023-12-15T16:54:39.560 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:54:39.564 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:54:39.564 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:54:40.612 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:54:40.613 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:54:49.690 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-test-debuginfo-19.0.0-157.g021ac167.el9.x8 14 MB/s | 623 MB 00:44 2023-12-15T16:54:49.691 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:54:49.691 INFO:teuthology.orchestra.run.smithi069.stdout:Total 14 MB/s | 623 MB 00:44 2023-12-15T16:54:49.691 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:54:49.697 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:54:49.697 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:54:53.366 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:54:53.366 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:54:58.329 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:54:58.421 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-157.g021ac167.el9.x8 1/1 2023-12-15T16:55:02.939 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-157.g021ac167.el9.x8 1/1 2023-12-15T16:55:03.211 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-157.g021ac167.el9.x8 1/1 2023-12-15T16:55:03.211 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:55:03.211 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:55:03.211 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:55:03.211 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:55:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:55:03.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test-debuginfo 2023-12-15T16:55:03.994 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:10 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:55:04.138 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:55:04.139 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:55:04.139 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:55:04.139 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:55:04.139 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:55:04.139 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 623 M 2023-12-15T16:55:04.139 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:55:04.139 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:55:04.139 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:55:04.139 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:55:04.140 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:55:04.140 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 623 M 2023-12-15T16:55:04.140 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.8 G 2023-12-15T16:55:04.140 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:55:38.631 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-test-debuginfo-19.0.0-157.g021ac167.el9.x8 18 MB/s | 623 MB 00:34 2023-12-15T16:55:38.632 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:55:38.632 INFO:teuthology.orchestra.run.smithi192.stdout:Total 18 MB/s | 623 MB 00:34 2023-12-15T16:55:38.632 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:55:38.637 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:55:38.637 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:55:42.318 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:55:42.318 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:55:58.482 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:55:59.031 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:56:07.267 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:56:07.544 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:56:07.545 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:07.545 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:56:07.545 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:56:07.545 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:07.545 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:56:07.802 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base-debuginfo 2023-12-15T16:56:08.353 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:20 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:56:08.446 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-base-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:56:08.494 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:56:08.495 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-12-15T16:56:08.495 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:56:08.538 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-debuginfo 2023-12-15T16:56:09.092 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:21 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:56:09.234 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:56:09.234 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:09.234 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:56:09.235 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:09.235 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:56:09.235 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 19 M 2023-12-15T16:56:09.235 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:09.235 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:56:09.235 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:09.235 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:56:09.235 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:09.235 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 19 M 2023-12-15T16:56:09.235 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 100 M 2023-12-15T16:56:09.235 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:56:10.551 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mgr-debuginfo-19.0.0-157.g021ac167.el9.x86 14 MB/s | 19 MB 00:01 2023-12-15T16:56:10.552 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:56:10.552 INFO:teuthology.orchestra.run.smithi069.stdout:Total 14 MB/s | 19 MB 00:01 2023-12-15T16:56:10.552 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:56:10.556 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:56:10.556 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:56:10.700 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:56:10.700 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:56:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:56:13.133 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:56:14.105 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:56:14.378 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:56:14.378 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:14.378 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:56:14.378 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:56:14.378 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:14.378 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:56:14.576 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mds-debuginfo 2023-12-15T16:56:15.129 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:27 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:56:15.270 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:56:15.271 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:15.271 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:56:15.271 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:15.271 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:56:15.272 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 47 M 2023-12-15T16:56:15.272 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:15.272 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:56:15.272 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:15.272 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:56:15.272 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:15.272 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 47 M 2023-12-15T16:56:15.272 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 168 M 2023-12-15T16:56:15.272 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:56:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mds-debuginfo-19.0.0-157.g021ac167.el9.x86 20 MB/s | 47 MB 00:02 2023-12-15T16:56:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:56:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:Total 20 MB/s | 47 MB 00:02 2023-12-15T16:56:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:56:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:56:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:56:17.898 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:56:17.899 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:56:22.487 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:56:22.676 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:56:23.999 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:56:24.269 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:56:24.269 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:24.269 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:56:24.269 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:56:24.270 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:24.270 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:56:24.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mon-debuginfo 2023-12-15T16:56:25.006 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:37 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:56:25.148 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:56:25.149 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:25.149 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:56:25.149 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:25.149 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:56:25.150 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 111 M 2023-12-15T16:56:25.150 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:25.150 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:56:25.150 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:25.150 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:56:25.150 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:25.150 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 111 M 2023-12-15T16:56:25.150 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 507 M 2023-12-15T16:56:25.150 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:56:30.580 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mon-debuginfo-19.0.0-157.g021ac167.el9.x86 21 MB/s | 111 MB 00:05 2023-12-15T16:56:30.581 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:56:30.582 INFO:teuthology.orchestra.run.smithi069.stdout:Total 20 MB/s | 111 MB 00:05 2023-12-15T16:56:30.582 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:56:30.586 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:56:30.586 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:56:31.269 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:56:31.269 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:56:42.829 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:56:42.927 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:56:46.074 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:56:46.347 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:56:46.347 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:46.347 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:56:46.347 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:56:46.347 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:46.347 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:56:46.541 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-osd-debuginfo 2023-12-15T16:56:47.097 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:59 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:56:47.239 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:56:47.239 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 311 M 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 311 M 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.3 G 2023-12-15T16:56:47.240 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:56:47.654 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:56:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:56:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:56:57.366 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:56:57.367 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:56:57.367 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:56:57.367 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:56:57.367 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:56:57.367 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:56:57.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base-debuginfo 2023-12-15T16:56:58.183 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:06:05 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:56:58.277 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 is already installed. 2023-12-15T16:56:58.326 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:56:58.327 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-12-15T16:56:58.328 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:56:58.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-debuginfo 2023-12-15T16:56:58.936 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:06:05 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:56:59.080 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:56:59.081 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:56:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:56:59.081 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:56:59.082 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:56:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 19 M 2023-12-15T16:56:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:56:59.082 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:56:59.082 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:56:59.082 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:56:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:56:59.082 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 19 M 2023-12-15T16:56:59.082 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 100 M 2023-12-15T16:56:59.082 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:57:01.455 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-osd-debuginfo-19.0.0-157.g021ac167.el9.x86 22 MB/s | 311 MB 00:14 2023-12-15T16:57:01.456 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:57:01.456 INFO:teuthology.orchestra.run.smithi069.stdout:Total 22 MB/s | 311 MB 00:14 2023-12-15T16:57:01.456 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:57:01.460 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:57:01.461 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:57:03.303 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:57:03.303 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:57:06.074 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-mgr-debuginfo-19.0.0-157.g021ac167.el9.x86 2.7 MB/s | 19 MB 00:06 2023-12-15T16:57:06.075 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:57:06.075 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.7 MB/s | 19 MB 00:06 2023-12-15T16:57:06.075 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:57:06.079 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:57:06.080 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:57:06.226 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:57:06.226 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:57:08.397 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:57:08.572 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:09.606 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:09.863 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:09.863 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:09.863 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:57:09.863 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:57:09.863 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:09.864 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:57:10.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mds-debuginfo 2023-12-15T16:57:10.616 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:06:17 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:57:10.760 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:57:10.760 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:57:10.760 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:57:10.760 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:57:10.761 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:57:10.761 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 47 M 2023-12-15T16:57:10.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:10.761 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:57:10.761 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:57:10.761 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:57:10.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:10.761 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 47 M 2023-12-15T16:57:10.761 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 168 M 2023-12-15T16:57:10.761 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:57:15.009 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-mds-debuginfo-19.0.0-157.g021ac167.el9.x86 11 MB/s | 47 MB 00:04 2023-12-15T16:57:15.010 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:57:15.010 INFO:teuthology.orchestra.run.smithi192.stdout:Total 11 MB/s | 47 MB 00:04 2023-12-15T16:57:15.011 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:57:15.015 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:57:15.015 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:57:15.320 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:57:15.320 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:57:19.944 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:57:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:21.441 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:21.712 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:21.712 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:21.712 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:57:21.712 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:57:21.712 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:21.712 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:57:21.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mon-debuginfo 2023-12-15T16:57:22.457 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:06:29 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:57:22.600 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:57:22.600 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:57:22.600 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:57:22.600 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:57:22.601 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:57:22.601 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 111 M 2023-12-15T16:57:22.601 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:22.601 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:57:22.601 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:57:22.601 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:57:22.601 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:22.601 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 111 M 2023-12-15T16:57:22.601 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 507 M 2023-12-15T16:57:22.601 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:57:35.060 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:57:35.207 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:40.127 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-mon-debuginfo-19.0.0-157.g021ac167.el9.x86 6.4 MB/s | 111 MB 00:17 2023-12-15T16:57:40.127 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:57:40.128 INFO:teuthology.orchestra.run.smithi192.stdout:Total 6.4 MB/s | 111 MB 00:17 2023-12-15T16:57:40.128 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:57:40.132 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:57:40.132 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:57:40.815 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:57:40.816 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:57:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:43.421 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:43.422 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:43.422 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:57:43.422 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:57:43.422 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:43.422 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:57:43.643 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-fuse-debuginfo 2023-12-15T16:57:44.199 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:07:56 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:57:44.344 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:57:44.345 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:44.345 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:57:44.345 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:44.345 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:57:44.345 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 5.8 M 2023-12-15T16:57:44.345 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:44.345 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:57:44.345 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:44.345 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:57:44.345 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:44.346 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 5.8 M 2023-12-15T16:57:44.346 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 28 M 2023-12-15T16:57:44.346 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:57:45.103 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-fuse-debuginfo-19.0.0-157.g021ac167.el9.x8 7.6 MB/s | 5.8 MB 00:00 2023-12-15T16:57:45.104 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:57:45.104 INFO:teuthology.orchestra.run.smithi069.stdout:Total 7.6 MB/s | 5.8 MB 00:00 2023-12-15T16:57:45.104 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:57:45.108 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:57:45.108 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:57:45.174 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:57:45.174 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:57:45.887 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:57:45.984 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:57:46.549 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:57:46.838 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:57:46.838 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:46.838 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:57:46.839 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:57:46.839 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:46.839 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:57:47.015 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados-devel-debuginfo 2023-12-15T16:57:47.566 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:07:59 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:57:47.710 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:57:47.710 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:47.710 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:57:47.710 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:47.710 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:57:47.711 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 108 k 2023-12-15T16:57:47.711 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:47.711 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:57:47.711 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:47.711 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:57:47.711 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:47.711 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 108 k 2023-12-15T16:57:47.711 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 440 k 2023-12-15T16:57:47.711 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:57:48.031 INFO:teuthology.orchestra.run.smithi069.stdout:librados-devel-debuginfo-19.0.0-157.g021ac167.e 339 kB/s | 108 kB 00:00 2023-12-15T16:57:48.031 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:57:48.032 INFO:teuthology.orchestra.run.smithi069.stdout:Total 338 kB/s | 108 kB 00:00 2023-12-15T16:57:48.032 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:57:48.035 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:57:48.035 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:57:48.069 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:57:48.070 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:57:48.188 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:57:48.279 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:57:48.663 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:57:48.937 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:57:48.938 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:48.938 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:57:48.938 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:57:48.938 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:48.938 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:57:49.139 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs2-debuginfo 2023-12-15T16:57:49.693 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:01 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:57:49.835 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:57:49.836 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:49.836 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:57:49.836 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:49.836 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:57:49.836 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 5.1 M 2023-12-15T16:57:49.836 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:49.836 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:57:49.836 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:49.837 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:57:49.837 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:49.837 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 5.1 M 2023-12-15T16:57:49.837 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 24 M 2023-12-15T16:57:49.837 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:57:50.784 INFO:teuthology.orchestra.run.smithi069.stdout:libcephfs2-debuginfo-19.0.0-157.g021ac167.el9.x 5.4 MB/s | 5.1 MB 00:00 2023-12-15T16:57:50.785 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:57:50.785 INFO:teuthology.orchestra.run.smithi069.stdout:Total 5.4 MB/s | 5.1 MB 00:00 2023-12-15T16:57:50.785 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:57:50.789 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:57:50.789 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:57:50.852 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:57:50.852 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:57:51.493 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:57:51.598 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T16:57:52.113 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T16:57:52.387 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T16:57:52.387 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:52.387 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:57:52.387 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:57:52.387 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:52.387 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:57:52.392 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:57:52.523 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:52.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados2-debuginfo 2023-12-15T16:57:53.144 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:05 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:57:53.287 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:57:53.288 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:53.288 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:57:53.288 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:53.288 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:57:53.288 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 43 M 2023-12-15T16:57:53.288 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:53.288 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:57:53.288 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:57:53.288 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:57:53.289 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:57:53.289 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 43 M 2023-12-15T16:57:53.289 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 248 M 2023-12-15T16:57:53.289 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:57:55.637 INFO:teuthology.orchestra.run.smithi069.stdout:librados2-debuginfo-19.0.0-157.g021ac167.el9.x8 18 MB/s | 43 MB 00:02 2023-12-15T16:57:55.638 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:57:55.638 INFO:teuthology.orchestra.run.smithi069.stdout:Total 18 MB/s | 43 MB 00:02 2023-12-15T16:57:55.638 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:57:55.642 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:57:55.642 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:57:55.718 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:55.937 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:57:55.937 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:57:55.985 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:57:55.985 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:55.985 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:57:55.985 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:57:55.985 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:55.985 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:57:56.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-osd-debuginfo 2023-12-15T16:57:56.760 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:07:03 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:57:56.904 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:57:56.904 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:57:56.904 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:57:56.904 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:57:56.904 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:57:56.904 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 311 M 2023-12-15T16:57:56.904 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:56.905 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:57:56.905 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:57:56.905 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:57:56.905 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:57:56.905 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 311 M 2023-12-15T16:57:56.905 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.3 G 2023-12-15T16:57:56.905 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:58:00.853 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:58:00.951 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:58:02.748 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:58:03.018 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:58:03.018 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:03.018 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:58:03.018 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:03.018 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:03.018 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:58:03.249 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librbd1-debuginfo 2023-12-15T16:58:03.800 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:15 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:58:03.944 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 50 M 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 50 M 2023-12-15T16:58:03.945 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 294 M 2023-12-15T16:58:03.946 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:58:11.125 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-osd-debuginfo-19.0.0-157.g021ac167.el9.x86 22 MB/s | 311 MB 00:14 2023-12-15T16:58:11.126 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:58:11.126 INFO:teuthology.orchestra.run.smithi192.stdout:Total 22 MB/s | 311 MB 00:14 2023-12-15T16:58:11.127 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:58:11.131 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:58:11.131 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:58:12.979 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:58:12.979 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:58:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:librbd1-debuginfo-19.0.0-157.g021ac167.el9.x86_ 3.7 MB/s | 50 MB 00:13 2023-12-15T16:58:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:58:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:Total 3.7 MB/s | 50 MB 00:13 2023-12-15T16:58:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:58:17.318 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:58:17.318 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:58:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:58:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:58:23.372 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:58:23.468 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librbd1-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:25.617 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:25.889 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:25.889 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:25.889 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:58:25.889 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:25.889 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:25.889 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:58:26.076 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-cephfs-debuginfo 2023-12-15T16:58:26.627 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:38 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:58:26.771 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:58:26.772 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:26.772 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:58:26.772 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:26.772 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:58:26.772 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 615 k 2023-12-15T16:58:26.772 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:26.772 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:58:26.772 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:26.772 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:58:26.772 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:26.772 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 615 k 2023-12-15T16:58:26.773 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.1 M 2023-12-15T16:58:26.773 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:58:27.074 INFO:teuthology.orchestra.run.smithi069.stdout:python3-cephfs-debuginfo-19.0.0-157.g021ac167.e 2.0 MB/s | 615 kB 00:00 2023-12-15T16:58:27.074 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:58:27.074 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.0 MB/s | 615 kB 00:00 2023-12-15T16:58:27.075 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:58:27.078 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:58:27.078 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:58:27.114 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:58:27.115 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:58:27.281 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:58:27.381 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:58:27.791 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:58:28.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:58:28.089 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:28.089 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:58:28.089 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:28.089 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:28.089 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:58:28.271 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rados-debuginfo 2023-12-15T16:58:28.826 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:40 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:58:28.969 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:58:28.970 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:28.970 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:58:28.970 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:28.970 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:58:28.970 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 770 k 2023-12-15T16:58:28.971 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:28.971 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:58:28.971 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:28.971 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:58:28.971 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:28.971 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 770 k 2023-12-15T16:58:28.971 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.7 M 2023-12-15T16:58:28.971 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:58:29.410 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rados-debuginfo-19.0.0-157.g021ac167.el 1.7 MB/s | 770 kB 00:00 2023-12-15T16:58:29.410 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:58:29.410 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.7 MB/s | 770 kB 00:00 2023-12-15T16:58:29.411 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:58:29.414 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:58:29.414 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:58:29.453 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:58:29.453 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:58:29.631 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:58:29.734 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-debuginfo-2:19.0.0-157.g021ac167.el9.x 1/1 2023-12-15T16:58:30.106 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-157.g021ac167.el9.x 1/1 2023-12-15T16:58:30.374 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-157.g021ac167.el9.x 1/1 2023-12-15T16:58:30.374 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:30.374 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:58:30.374 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:30.374 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:30.374 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:58:30.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rbd-debuginfo 2023-12-15T16:58:31.122 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:43 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:58:31.266 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:58:31.266 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 1.1 M 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.1 M 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 4.3 M 2023-12-15T16:58:31.267 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:58:31.738 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rbd-debuginfo-19.0.0-157.g021ac167.el9. 2.4 MB/s | 1.1 MB 00:00 2023-12-15T16:58:31.738 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:58:31.738 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.4 MB/s | 1.1 MB 00:00 2023-12-15T16:58:31.738 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:58:31.742 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:58:31.742 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:58:31.782 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:58:31.782 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:58:32.000 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:58:32.119 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:58:32.520 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:58:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:58:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:32.797 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:58:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:32.797 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:58:33.022 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rgw-debuginfo 2023-12-15T16:58:33.577 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:45 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:58:33.721 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:58:33.722 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:33.722 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:58:33.722 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:33.722 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:58:33.722 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 337 k 2023-12-15T16:58:33.722 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:33.722 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:58:33.722 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:33.722 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:58:33.722 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:33.723 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 337 k 2023-12-15T16:58:33.723 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.0 M 2023-12-15T16:58:33.723 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:58:33.991 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rgw-debuginfo-19.0.0-157.g021ac167.el9. 1.2 MB/s | 337 kB 00:00 2023-12-15T16:58:33.991 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:58:33.991 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.2 MB/s | 337 kB 00:00 2023-12-15T16:58:33.992 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:58:33.995 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:58:33.995 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:58:34.030 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:58:34.030 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:58:34.169 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:58:34.265 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:58:34.640 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:58:34.915 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:58:34.915 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:34.915 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:58:34.915 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:34.915 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:34.915 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:58:35.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-fuse-debuginfo 2023-12-15T16:58:35.662 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:47 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:58:35.805 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:58:35.805 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:35.805 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:58:35.805 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:35.806 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:58:35.806 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 506 k 2023-12-15T16:58:35.806 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:35.806 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:58:35.806 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:35.806 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:58:35.806 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:35.806 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 506 k 2023-12-15T16:58:35.806 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.5 M 2023-12-15T16:58:35.806 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:58:36.077 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-fuse-debuginfo-19.0.0-157.g021ac167.el9.x86 1.8 MB/s | 506 kB 00:00 2023-12-15T16:58:36.078 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:58:36.078 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.8 MB/s | 506 kB 00:00 2023-12-15T16:58:36.078 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:58:36.081 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:58:36.082 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:58:36.119 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:58:36.119 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:58:36.274 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:58:36.384 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:36.787 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:37.064 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:58:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:37.064 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:58:37.261 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-mirror-debuginfo 2023-12-15T16:58:37.812 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:49 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:58:37.954 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:58:37.955 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:37.955 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:58:37.955 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:37.955 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:58:37.955 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 50 M 2023-12-15T16:58:37.955 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:37.956 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:58:37.956 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:37.956 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:58:37.956 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:37.956 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 50 M 2023-12-15T16:58:37.956 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 284 M 2023-12-15T16:58:37.956 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:58:40.672 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-mirror-debuginfo-19.0.0-157.g021ac167.el9.x 18 MB/s | 50 MB 00:02 2023-12-15T16:58:40.673 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:58:40.673 INFO:teuthology.orchestra.run.smithi069.stdout:Total 18 MB/s | 50 MB 00:02 2023-12-15T16:58:40.673 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:58:40.677 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:58:40.678 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:58:41.003 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:58:41.003 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:58:44.966 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:58:45.129 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:46.710 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:58:46.812 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T16:58:48.817 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T16:58:49.095 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T16:58:49.095 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:49.095 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:58:49.095 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:49.095 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:49.095 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:58:49.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-nbd-debuginfo 2023-12-15T16:58:49.853 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:09:01 ago on Fri 15 Dec 2023 04:49:48 PM UTC. 2023-12-15T16:58:49.997 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-12-15T16:58:49.998 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:49.998 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-12-15T16:58:49.998 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:49.998 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-12-15T16:58:49.998 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 1.2 M 2023-12-15T16:58:49.998 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:49.998 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-12-15T16:58:49.999 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-12-15T16:58:49.999 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-12-15T16:58:49.999 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:49.999 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.2 M 2023-12-15T16:58:49.999 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 5.4 M 2023-12-15T16:58:49.999 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-12-15T16:58:50.307 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-nbd-debuginfo-19.0.0-157.g021ac167.el9.x86_ 4.0 MB/s | 1.2 MB 00:00 2023-12-15T16:58:50.307 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:58:50.308 INFO:teuthology.orchestra.run.smithi069.stdout:Total 4.0 MB/s | 1.2 MB 00:00 2023-12-15T16:58:50.308 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-12-15T16:58:50.311 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-12-15T16:58:50.311 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-12-15T16:58:50.352 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-12-15T16:58:50.352 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-12-15T16:58:50.581 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-12-15T16:58:50.681 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:51.086 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:51.361 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:51.361 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:51.361 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-12-15T16:58:51.361 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:51.362 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T16:58:51.362 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-12-15T16:58:51.575 DEBUG:teuthology.parallel:result is None 2023-12-15T16:58:53.281 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:53.560 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:58:53.560 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:53.560 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:58:53.560 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:53.561 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:53.561 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:58:53.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse-debuginfo 2023-12-15T16:58:54.341 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:01 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:58:54.486 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 5.8 M 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 5.8 M 2023-12-15T16:58:54.487 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 28 M 2023-12-15T16:58:54.488 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:58:54.999 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-fuse-debuginfo-19.0.0-157.g021ac167.el9.x8 11 MB/s | 5.8 MB 00:00 2023-12-15T16:58:55.000 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:58:55.000 INFO:teuthology.orchestra.run.smithi192.stdout:Total 11 MB/s | 5.8 MB 00:00 2023-12-15T16:58:55.000 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:58:55.004 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:58:55.004 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:58:55.072 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:58:55.072 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:58:55.781 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:58:55.889 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:58:56.430 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:58:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:58:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:56.691 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:58:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:56.692 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:58:56.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel-debuginfo 2023-12-15T16:58:57.446 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:04 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:58:57.590 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:58:57.591 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:58:57.591 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:58:57.591 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:58:57.591 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:58:57.591 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 108 k 2023-12-15T16:58:57.591 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:57.591 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:58:57.592 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:58:57.592 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:58:57.592 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:57.592 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 108 k 2023-12-15T16:58:57.592 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 440 k 2023-12-15T16:58:57.592 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:58:57.819 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-debuginfo-19.0.0-157.g021ac167.e 478 kB/s | 108 kB 00:00 2023-12-15T16:58:57.819 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:58:57.819 INFO:teuthology.orchestra.run.smithi192.stdout:Total 475 kB/s | 108 kB 00:00 2023-12-15T16:58:57.820 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:58:57.823 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:58:57.823 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:58:57.857 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:58:57.857 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:58:57.970 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:58:58.068 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:58:58.484 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:58:58.751 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:58:58.751 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:58.751 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:58:58.751 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:58:58.751 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:58.751 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:58:58.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2-debuginfo 2023-12-15T16:58:59.509 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:06 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:58:59.652 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:58:59.653 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:58:59.653 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:58:59.653 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:58:59.653 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:58:59.653 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 5.1 M 2023-12-15T16:58:59.653 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:59.653 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:58:59.653 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:58:59.653 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:58:59.653 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:58:59.653 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 5.1 M 2023-12-15T16:58:59.654 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 24 M 2023-12-15T16:58:59.654 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:59:01.113 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs2-debuginfo-19.0.0-157.g021ac167.el9.x 3.5 MB/s | 5.1 MB 00:01 2023-12-15T16:59:01.114 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:59:01.114 INFO:teuthology.orchestra.run.smithi192.stdout:Total 3.5 MB/s | 5.1 MB 00:01 2023-12-15T16:59:01.115 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:59:01.119 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:59:01.119 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:59:01.185 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:59:01.185 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:59:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:59:01.920 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T16:59:02.478 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T16:59:02.764 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T16:59:02.764 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:02.764 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:59:02.764 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:59:02.764 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:02.764 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:59:02.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2-debuginfo 2023-12-15T16:59:03.523 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:10 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:59:03.667 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:59:03.667 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:03.668 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:59:03.668 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:03.668 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:59:03.668 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 43 M 2023-12-15T16:59:03.668 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:03.668 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:59:03.668 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:03.668 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:59:03.668 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:03.669 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 43 M 2023-12-15T16:59:03.669 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 248 M 2023-12-15T16:59:03.669 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:59:09.149 INFO:teuthology.orchestra.run.smithi192.stdout:librados2-debuginfo-19.0.0-157.g021ac167.el9.x8 7.9 MB/s | 43 MB 00:05 2023-12-15T16:59:09.150 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:59:09.150 INFO:teuthology.orchestra.run.smithi192.stdout:Total 7.9 MB/s | 43 MB 00:05 2023-12-15T16:59:09.150 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:59:09.155 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:59:09.155 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:59:09.450 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:59:09.450 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:59:14.378 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:59:14.463 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:59:16.332 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:59:16.605 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_6 1/1 2023-12-15T16:59:16.606 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:16.606 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:59:16.606 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:59:16.606 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:16.606 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:59:16.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1-debuginfo 2023-12-15T16:59:17.367 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:24 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:59:17.511 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:59:17.512 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:17.512 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:59:17.512 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:17.512 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:59:17.512 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 50 M 2023-12-15T16:59:17.512 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:17.513 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:59:17.513 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:17.513 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:59:17.513 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:17.513 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 50 M 2023-12-15T16:59:17.513 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 294 M 2023-12-15T16:59:17.513 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:59:24.151 INFO:teuthology.orchestra.run.smithi192.stdout:librbd1-debuginfo-19.0.0-157.g021ac167.el9.x86_ 7.5 MB/s | 50 MB 00:06 2023-12-15T16:59:24.152 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:59:24.152 INFO:teuthology.orchestra.run.smithi192.stdout:Total 7.5 MB/s | 50 MB 00:06 2023-12-15T16:59:24.153 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:59:24.157 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:59:24.157 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:59:24.491 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:59:24.491 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:59:30.228 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:59:30.315 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librbd1-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:59:32.431 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:59:32.716 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:59:32.716 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:32.716 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:59:32.716 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:59:32.716 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:32.716 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:59:32.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs-debuginfo 2023-12-15T16:59:33.468 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:40 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:59:33.613 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:59:33.614 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:33.614 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:59:33.614 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:33.614 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:59:33.614 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 615 k 2023-12-15T16:59:33.614 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:33.614 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:59:33.614 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:33.614 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:59:33.614 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:33.615 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 615 k 2023-12-15T16:59:33.615 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.1 M 2023-12-15T16:59:33.615 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:59:34.495 INFO:teuthology.orchestra.run.smithi192.stdout:python3-cephfs-debuginfo-19.0.0-157.g021ac167.e 699 kB/s | 615 kB 00:00 2023-12-15T16:59:34.495 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:59:34.496 INFO:teuthology.orchestra.run.smithi192.stdout:Total 698 kB/s | 615 kB 00:00 2023-12-15T16:59:34.496 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:59:34.500 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:59:34.500 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:59:34.537 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:59:34.538 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:59:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:59:34.793 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:59:35.202 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:59:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-157.g021ac167.el9. 1/1 2023-12-15T16:59:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:35.488 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:59:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:59:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:35.489 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:59:35.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados-debuginfo 2023-12-15T16:59:36.235 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:43 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:59:36.380 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:59:36.381 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:36.381 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:59:36.381 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:36.381 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:59:36.381 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 770 k 2023-12-15T16:59:36.382 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:36.382 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:59:36.382 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:36.382 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:59:36.382 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:36.382 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 770 k 2023-12-15T16:59:36.382 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.7 M 2023-12-15T16:59:36.382 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:59:37.447 INFO:teuthology.orchestra.run.smithi192.stdout:python3-rados-debuginfo-19.0.0-157.g021ac167.el 723 kB/s | 770 kB 00:01 2023-12-15T16:59:37.448 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:59:37.448 INFO:teuthology.orchestra.run.smithi192.stdout:Total 722 kB/s | 770 kB 00:01 2023-12-15T16:59:37.448 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:59:37.452 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:59:37.452 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:59:37.491 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:59:37.491 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:59:37.673 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:59:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-debuginfo-2:19.0.0-157.g021ac167.el9.x 1/1 2023-12-15T16:59:38.185 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-157.g021ac167.el9.x 1/1 2023-12-15T16:59:38.442 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-157.g021ac167.el9.x 1/1 2023-12-15T16:59:38.443 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:38.443 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:59:38.443 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:59:38.443 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:38.443 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:59:38.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd-debuginfo 2023-12-15T16:59:39.201 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:46 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:59:39.344 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:59:39.345 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:39.345 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:59:39.345 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:39.346 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:59:39.346 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 1.1 M 2023-12-15T16:59:39.346 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:39.346 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:59:39.346 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:39.346 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:59:39.346 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:39.346 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.1 M 2023-12-15T16:59:39.346 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 4.3 M 2023-12-15T16:59:39.346 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:59:40.726 INFO:teuthology.orchestra.run.smithi192.stdout:python3-rbd-debuginfo-19.0.0-157.g021ac167.el9. 835 kB/s | 1.1 MB 00:01 2023-12-15T16:59:40.726 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:59:40.727 INFO:teuthology.orchestra.run.smithi192.stdout:Total 834 kB/s | 1.1 MB 00:01 2023-12-15T16:59:40.727 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:59:40.731 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:59:40.731 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:59:40.774 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:59:40.775 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:59:40.993 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:59:41.098 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:59:41.491 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:59:41.785 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:59:41.786 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:41.786 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:59:41.786 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:59:41.786 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:41.786 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:59:42.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw-debuginfo 2023-12-15T16:59:42.574 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:49 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:59:42.719 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:59:42.720 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:42.720 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:59:42.720 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:42.721 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:59:42.721 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 337 k 2023-12-15T16:59:42.721 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:42.721 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:59:42.721 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:42.721 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:59:42.721 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:42.721 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 337 k 2023-12-15T16:59:42.721 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.0 M 2023-12-15T16:59:42.721 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:59:43.578 INFO:teuthology.orchestra.run.smithi192.stdout:python3-rgw-debuginfo-19.0.0-157.g021ac167.el9. 393 kB/s | 337 kB 00:00 2023-12-15T16:59:43.579 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:59:43.580 INFO:teuthology.orchestra.run.smithi192.stdout:Total 392 kB/s | 337 kB 00:00 2023-12-15T16:59:43.580 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:59:43.583 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:59:43.583 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:59:43.621 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:59:43.622 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:59:43.762 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:59:43.861 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:59:44.262 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:59:44.546 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-157.g021ac167.el9.x86 1/1 2023-12-15T16:59:44.546 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:44.546 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:59:44.547 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:59:44.547 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:44.547 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:59:44.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse-debuginfo 2023-12-15T16:59:45.316 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:52 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:59:45.460 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:59:45.461 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:45.461 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:59:45.461 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:45.461 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:59:45.461 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 506 k 2023-12-15T16:59:45.461 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:45.461 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:59:45.461 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:45.461 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:59:45.462 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:45.462 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 506 k 2023-12-15T16:59:45.462 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.5 M 2023-12-15T16:59:45.462 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T16:59:46.357 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-debuginfo-19.0.0-157.g021ac167.el9.x86 565 kB/s | 506 kB 00:00 2023-12-15T16:59:46.358 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T16:59:46.358 INFO:teuthology.orchestra.run.smithi192.stdout:Total 564 kB/s | 506 kB 00:00 2023-12-15T16:59:46.358 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T16:59:46.362 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T16:59:46.362 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T16:59:46.401 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T16:59:46.402 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T16:59:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T16:59:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:59:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:59:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T16:59:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:47.304 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T16:59:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T16:59:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:47.304 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T16:59:47.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror-debuginfo 2023-12-15T16:59:48.063 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:55 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T16:59:48.209 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T16:59:48.209 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 50 M 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 50 M 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 284 M 2023-12-15T16:59:48.210 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T17:00:02.967 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-debuginfo-19.0.0-157.g021ac167.el9.x 3.4 MB/s | 50 MB 00:14 2023-12-15T17:00:02.967 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T17:00:02.968 INFO:teuthology.orchestra.run.smithi192.stdout:Total 3.4 MB/s | 50 MB 00:14 2023-12-15T17:00:02.968 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T17:00:02.972 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T17:00:02.972 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T17:00:03.301 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T17:00:03.301 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T17:00:08.973 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T17:00:09.073 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T17:00:11.096 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T17:00:11.372 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-157.g021ac167.el9.x86_ 1/1 2023-12-15T17:00:11.372 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:11.372 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T17:00:11.372 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T17:00:11.372 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:11.372 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T17:00:11.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd-debuginfo 2023-12-15T17:00:12.130 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:09:19 ago on Fri 15 Dec 2023 04:50:53 PM UTC. 2023-12-15T17:00:12.274 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-15T17:00:12.275 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T17:00:12.275 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-12-15T17:00:12.275 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T17:00:12.275 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-15T17:00:12.275 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-157.g021ac167.el9 ceph 1.2 M 2023-12-15T17:00:12.276 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:12.276 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-15T17:00:12.276 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-15T17:00:12.276 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-12-15T17:00:12.276 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:12.276 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.2 M 2023-12-15T17:00:12.276 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 5.4 M 2023-12-15T17:00:12.276 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-15T17:00:12.587 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-debuginfo-19.0.0-157.g021ac167.el9.x86_ 4.0 MB/s | 1.2 MB 00:00 2023-12-15T17:00:12.588 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-15T17:00:12.588 INFO:teuthology.orchestra.run.smithi192.stdout:Total 3.9 MB/s | 1.2 MB 00:00 2023-12-15T17:00:12.588 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-15T17:00:12.591 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-15T17:00:12.592 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-15T17:00:12.632 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-15T17:00:12.632 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-15T17:00:12.861 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-15T17:00:12.969 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T17:00:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T17:00:13.629 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 1/1 2023-12-15T17:00:13.629 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:13.629 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-15T17:00:13.629 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-debuginfo-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-15T17:00:13.629 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:13.629 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-15T17:00:13.856 DEBUG:teuthology.parallel:result is None 2023-12-15T17:00:13.856 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-15T17:00:14.076 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-15T17:00:14.101 INFO:teuthology.orchestra.run.smithi069.stdout:19.0.0-157.g021ac167.el9 2023-12-15T17:00:14.101 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-157.g021ac167.el9 2023-12-15T17:00:14.102 INFO:teuthology.task.install:The correct ceph version 19.0.0-157.g021ac167 is installed. 2023-12-15T17:00:14.104 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-15T17:00:14.320 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-15T17:00:14.342 INFO:teuthology.orchestra.run.smithi192.stdout:19.0.0-157.g021ac167.el9 2023-12-15T17:00:14.342 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-157.g021ac167.el9 2023-12-15T17:00:14.342 INFO:teuthology.task.install:The correct ceph version 19.0.0-157.g021ac167 is installed. 2023-12-15T17:00:14.344 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-15T17:00:14.344 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:14.344 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-15T17:00:14.376 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:14.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-15T17:00:14.411 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-15T17:00:14.411 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:14.411 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2023-12-15T17:00:14.446 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-15T17:00:14.516 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:14.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2023-12-15T17:00:14.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-15T17:00:14.612 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-15T17:00:14.613 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:14.613 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-15T17:00:14.642 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-15T17:00:14.709 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:14.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-15T17:00:14.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-15T17:00:14.805 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-15T17:00:14.805 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:14.805 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2023-12-15T17:00:14.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-15T17:00:14.899 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:14.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2023-12-15T17:00:14.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-15T17:00:14.995 INFO:teuthology.run_tasks:Running task ceph... 2023-12-15T17:00:15.064 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-15T17:00:15.064 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2023-12-15T17:00:15.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /var/log/ceph 2023-12-15T17:00:15.095 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-15T17:00:15.095 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-15T17:00:15.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-15T17:00:15.162 INFO:tasks.ceph:Creating extra log directories... 2023-12-15T17:00:15.162 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-15T17:00:15.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-15T17:00:15.240 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-15T17:00:15.240 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-15T17:00:15.240 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262', 'branch': 'wip-yuri10-testing-2023-12-12-1229', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7493262', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri10-testing-2023-12-12-1229'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri10-testing-2023-12-12-1229', 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175'}}, 'owner': 'scheduled_lflores@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9963, 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri10-testing-2023-12-12-1229', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '021ac1670a21c19759080c6da90baaeb42e7d175', 'targets': {'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC2/eNabySf0IuZlE+BHIHvM/dceKYYIFmCN2lkbZT9YqRA6QeS/zPOLyRDI83a0zpFobrC9ZbJzDK85o4PM83k=', 'smithi192.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJSe1/+NcQykE1SfbIJYGnBvdauZOKvJs4f8q5eTJzeX4G6JG/f5bVREqsRWL1z0Szf90F/G1grP+vyeGpIPDBk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-15_16:16:51', 'tube': 'smithi', 'user': 'lflores', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-15T17:00:15.240 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-15T17:00:15.268 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-15T17:00:15.284 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2023-12-15T17:00:15.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/run/ceph 2023-12-15T17:00:15.351 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:15.351 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2023-12-15T17:00:15.403 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-15T17:00:15.404 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2023-12-15T17:00:15.458 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-15T17:00:15.458 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T17:00:15.458 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-12-15T17:00:15.458 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T17:00:15.458 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T17:00:15.458 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-12-15 16:51:25.917355142 +0000 2023-12-15T17:00:15.458 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-12-15 16:48:49.847955795 +0000 2023-12-15T17:00:15.459 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-12-15 16:48:49.847955795 +0000 2023-12-15T17:00:15.459 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2023-12-15 16:48:49.847955795 +0000 2023-12-15T17:00:15.459 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-15T17:00:15.523 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-12-15T17:00:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-12-15T17:00:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000188902 s, 2.7 MB/s 2023-12-15T17:00:15.525 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-15T17:00:15.580 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2023-12-15T17:00:15.634 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-15T17:00:15.634 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T17:00:15.634 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-12-15T17:00:15.634 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T17:00:15.634 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T17:00:15.634 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-12-15 16:51:25.917355142 +0000 2023-12-15T17:00:15.634 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-12-15 16:48:50.348944518 +0000 2023-12-15T17:00:15.634 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-12-15 16:48:50.348944518 +0000 2023-12-15T17:00:15.634 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2023-12-15 16:48:50.348944518 +0000 2023-12-15T17:00:15.635 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-15T17:00:15.699 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-12-15T17:00:15.699 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-12-15T17:00:15.699 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000237124 s, 2.2 MB/s 2023-12-15T17:00:15.700 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-15T17:00:15.757 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2023-12-15T17:00:15.811 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-15T17:00:15.811 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T17:00:15.811 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-15T17:00:15.811 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T17:00:15.811 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T17:00:15.811 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-12-15 16:51:25.917355142 +0000 2023-12-15T17:00:15.811 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-12-15 16:48:50.847933285 +0000 2023-12-15T17:00:15.811 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-12-15 16:48:50.847933285 +0000 2023-12-15T17:00:15.812 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2023-12-15 16:48:50.847933285 +0000 2023-12-15T17:00:15.812 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-15T17:00:15.877 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-12-15T17:00:15.878 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-12-15T17:00:15.878 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000197021 s, 2.6 MB/s 2023-12-15T17:00:15.879 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-15T17:00:15.934 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2023-12-15T17:00:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-15T17:00:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T17:00:15.989 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-12-15T17:00:15.989 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T17:00:15.990 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T17:00:15.990 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-12-15 16:51:25.917355142 +0000 2023-12-15T17:00:15.990 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-12-15 16:48:51.345922075 +0000 2023-12-15T17:00:15.990 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-12-15 16:48:51.345922075 +0000 2023-12-15T17:00:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2023-12-15 16:48:51.345922075 +0000 2023-12-15T17:00:15.990 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-15T17:00:16.055 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-12-15T17:00:16.055 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-12-15T17:00:16.055 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000187261 s, 2.7 MB/s 2023-12-15T17:00:16.056 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-15T17:00:16.112 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-15T17:00:16.112 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:16.112 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2023-12-15T17:00:16.130 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-15T17:00:16.130 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2023-12-15T17:00:16.187 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-15T17:00:16.187 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T17:00:16.187 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-12-15T17:00:16.188 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T17:00:16.188 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T17:00:16.188 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-12-15 16:52:46.392710426 +0000 2023-12-15T17:00:16.188 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-12-15 16:48:49.755735692 +0000 2023-12-15T17:00:16.188 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-12-15 16:48:49.755735692 +0000 2023-12-15T17:00:16.188 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2023-12-15 16:48:49.755735692 +0000 2023-12-15T17:00:16.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-15T17:00:16.256 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-12-15T17:00:16.256 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-12-15T17:00:16.256 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000219687 s, 2.3 MB/s 2023-12-15T17:00:16.257 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-15T17:00:16.315 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2023-12-15T17:00:16.371 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-15T17:00:16.371 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T17:00:16.371 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-15T17:00:16.371 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T17:00:16.372 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T17:00:16.372 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-12-15 16:52:46.392710426 +0000 2023-12-15T17:00:16.372 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-12-15 16:48:50.261725740 +0000 2023-12-15T17:00:16.372 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-12-15 16:48:50.261725740 +0000 2023-12-15T17:00:16.372 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2023-12-15 16:48:50.261725740 +0000 2023-12-15T17:00:16.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-15T17:00:16.439 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-12-15T17:00:16.440 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-12-15T17:00:16.440 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000208947 s, 2.5 MB/s 2023-12-15T17:00:16.441 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-15T17:00:16.499 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2023-12-15T17:00:16.555 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-15T17:00:16.555 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T17:00:16.555 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-12-15T17:00:16.555 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T17:00:16.556 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T17:00:16.556 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-12-15 16:52:46.392710426 +0000 2023-12-15T17:00:16.556 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-12-15 16:48:50.759715945 +0000 2023-12-15T17:00:16.556 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-12-15 16:48:50.759715945 +0000 2023-12-15T17:00:16.556 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2023-12-15 16:48:50.759715945 +0000 2023-12-15T17:00:16.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-15T17:00:16.622 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-12-15T17:00:16.622 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-12-15T17:00:16.622 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000197046 s, 2.6 MB/s 2023-12-15T17:00:16.623 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-15T17:00:16.680 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2023-12-15T17:00:16.736 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-15T17:00:16.736 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T17:00:16.736 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-12-15T17:00:16.736 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T17:00:16.736 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T17:00:16.736 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-12-15 16:52:46.393710406 +0000 2023-12-15T17:00:16.736 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-12-15 16:48:51.265705993 +0000 2023-12-15T17:00:16.736 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-12-15 16:48:51.265705993 +0000 2023-12-15T17:00:16.737 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2023-12-15 16:48:51.265705993 +0000 2023-12-15T17:00:16.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-15T17:00:16.803 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-12-15T17:00:16.803 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-12-15T17:00:16.803 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000167954 s, 3.0 MB/s 2023-12-15T17:00:16.804 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-15T17:00:16.860 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-15T17:00:16.860 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-15T17:00:16.860 INFO:tasks.ceph:Generating config... 2023-12-15T17:00:16.861 INFO:tasks.ceph:[client] debug monc = 20 2023-12-15T17:00:16.861 INFO:tasks.ceph:[client] debug ms = 1 2023-12-15T17:00:16.861 INFO:tasks.ceph:[client] debug objecter = 20 2023-12-15T17:00:16.861 INFO:tasks.ceph:[client] debug rados = 20 2023-12-15T17:00:16.862 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-15T17:00:16.862 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-12-15T17:00:16.862 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-15T17:00:16.862 INFO:tasks.ceph:[global] ms type = async 2023-12-15T17:00:16.862 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-12-15T17:00:16.862 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-12-15T17:00:16.863 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-15T17:00:16.863 INFO:tasks.ceph:[osd] osd class default list = * 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd class load list = * 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-12-15T17:00:16.864 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-15T17:00:16.864 INFO:tasks.ceph:Setting up mon.a... 2023-12-15T17:00:16.864 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-15T17:00:16.914 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.keyring 2023-12-15T17:00:16.916 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-15T17:00:16.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-15T17:00:16.988 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.69'), ('mon.c', '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]'), ('mon.b', '172.21.15.192')] 2023-12-15T17:00:16.989 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790],172.21.15.192', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-12-15T17:00:16.990 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:16.990 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-15T17:00:17.043 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.69 --addv c '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]' --add b 172.21.15.192 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:17.127 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:17.128 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: generated fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:00:17.128 INFO:teuthology.orchestra.run.smithi069.stdout:setting min_mon_release = quincy 2023-12-15T17:00:17.128 INFO:teuthology.orchestra.run.smithi069.stdout:epoch 0 2023-12-15T17:00:17.128 INFO:teuthology.orchestra.run.smithi069.stdout:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:00:17.128 INFO:teuthology.orchestra.run.smithi069.stdout:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:00:17.128 INFO:teuthology.orchestra.run.smithi069.stdout:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:00:17.128 INFO:teuthology.orchestra.run.smithi069.stdout:min_mon_release 17 (quincy) 2023-12-15T17:00:17.128 INFO:teuthology.orchestra.run.smithi069.stdout:election_strategy: 1 2023-12-15T17:00:17.128 INFO:teuthology.orchestra.run.smithi069.stdout:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:00:17.129 INFO:teuthology.orchestra.run.smithi069.stdout:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:00:17.129 INFO:teuthology.orchestra.run.smithi069.stdout:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:00:17.129 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-15T17:00:17.130 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-15T17:00:17.184 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1a9f4b8c-183d-4435-93ef-5057f2b4aded... 2023-12-15T17:00:17.185 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-15T17:00:17.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.275 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2023-12-15T17:00:17.276 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2023-12-15T17:00:17.276 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.276 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-15T17:00:17.276 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2023-12-15T17:00:17.276 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.276 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2023-12-15T17:00:17.276 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-15T17:00:17.276 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.276 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2023-12-15T17:00:17.276 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790],172.21.15.192" 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: mon client directed command retry = 5 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: mon election default strategy = 1 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: ms inject socket failures = 5000 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: ms type = async 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: osd heartbeat grace = 80 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:00:17.278 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2023-12-15T17:00:17.279 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2023-12-15T17:00:17.280 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore zero block detection = True 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: debug monc = 20 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 10 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2023-12-15T17:00:17.281 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd class default list = * 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd class load list = * 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd client watch timeout = 120 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify cached snaps = True 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify missing on start = True 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd fast shutdown = False 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock override recovery settings = True 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd op queue = debug_random 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd op queue cut off = debug_random 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: osd shutdown pgref assert = True 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2023-12-15T17:00:17.282 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2023-12-15T17:00:17.283 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2023-12-15T17:00:17.284 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: mon min osdmap epochs = 50 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd crush smoke test = False 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune interval = 2 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune min = 15 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune txsize = 2 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: mon scrub interval = 300 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on pool no app = False 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: paxos service trim min = 10 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2023-12-15T17:00:17.285 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-15T17:00:17.286 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-15T17:00:17.286 INFO:teuthology.orchestra.run.smithi069.stdout: debug monc = 20 2023-12-15T17:00:17.286 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-12-15T17:00:17.286 INFO:teuthology.orchestra.run.smithi069.stdout: debug objecter = 20 2023-12-15T17:00:17.286 INFO:teuthology.orchestra.run.smithi069.stdout: debug rados = 20 2023-12-15T17:00:17.286 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2023-12-15T17:00:17.286 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2023-12-15T17:00:17.286 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2023-12-15T17:00:17.286 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2023-12-15T17:00:17.287 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2023-12-15T17:00:17.288 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2023-12-15T17:00:17.289 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = "172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790],172.21.15.192" 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: mon client directed command retry = 5 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: mon election default strategy = 1 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject socket failures = 5000 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: ms type = async 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: osd heartbeat grace = 80 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2023-12-15T17:00:17.290 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug trim objects = true 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.291 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: bdev async discard = True 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: bdev enable discard = True 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore allocator = bitmap 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore zero block detection = True 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 1 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 1 2023-12-15T17:00:17.292 INFO:teuthology.orchestra.run.smithi192.stdout: debug monc = 20 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 10 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: osd class default list = * 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: osd class load list = * 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: osd client watch timeout = 120 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify cached snaps = True 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify missing on start = True 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: osd fast shutdown = False 2023-12-15T17:00:17.293 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock override recovery settings = True 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: osd op queue = debug_random 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: osd op queue cut off = debug_random 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: osd shutdown pgref assert = True 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2023-12-15T17:00:17.294 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.295 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: mon min osdmap epochs = 50 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd crush smoke test = False 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: mon osdmap full prune interval = 2 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: mon osdmap full prune min = 15 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: mon osdmap full prune txsize = 2 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: mon scrub interval = 300 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on pool no app = False 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: paxos service trim min = 10 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-15T17:00:17.296 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-15T17:00:17.297 INFO:teuthology.orchestra.run.smithi192.stdout: debug monc = 20 2023-12-15T17:00:17.297 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-12-15T17:00:17.297 INFO:teuthology.orchestra.run.smithi192.stdout: debug objecter = 20 2023-12-15T17:00:17.297 INFO:teuthology.orchestra.run.smithi192.stdout: debug rados = 20 2023-12-15T17:00:17.297 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2023-12-15T17:00:17.297 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.c] 2023-12-15T17:00:17.297 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.b] 2023-12-15T17:00:17.298 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-15T17:00:17.298 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-15T17:00:17.394 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-15T17:00:17.394 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:17.394 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-15T17:00:17.408 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:17.408 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-15T17:00:17.463 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2023-12-15T17:00:17.463 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:17.463 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-15T17:00:17.463 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-15T17:00:17.542 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:17.542 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:17.597 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi192.front.sepia.ceph.com 2023-12-15T17:00:17.597 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:17.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-15T17:00:17.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-15T17:00:17.638 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:17.638 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:17.693 INFO:tasks.ceph:Setting up mon nodes... 2023-12-15T17:00:17.693 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-15T17:00:17.694 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-15T17:00:17.749 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-15T17:00:17.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-15T17:00:17.811 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-15T17:00:17.813 INFO:tasks.ceph:Setting up mds nodes... 2023-12-15T17:00:17.814 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-15T17:00:17.814 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-15T17:00:17.856 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-15T17:00:17.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-15T17:00:17.913 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-15T17:00:17.928 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-15T17:00:17.928 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-15T17:00:17.928 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-15T17:00:17.954 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-15T17:00:17.954 INFO:tasks.ceph:role: osd.0 2023-12-15T17:00:17.955 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2023-12-15T17:00:17.955 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-15T17:00:18.028 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T17:00:18.028 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T17:00:18.028 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T17:00:18.028 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T17:00:18.028 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T17:00:18.028 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-12-15T17:00:18.028 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T17:00:18.028 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T17:00:18.029 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T17:00:18.029 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T17:00:18.543 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-12-15T17:00:18.545 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-12-15T17:00:18.546 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-15T17:00:18.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-15T17:00:18.728 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-15T17:00:18.796 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-15T17:00:18.796 INFO:tasks.ceph:role: osd.1 2023-12-15T17:00:18.797 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2023-12-15T17:00:18.797 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-15T17:00:18.871 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T17:00:18.871 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T17:00:18.871 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T17:00:18.871 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T17:00:18.871 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T17:00:18.871 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-12-15T17:00:18.871 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T17:00:18.871 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T17:00:18.871 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T17:00:18.871 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T17:00:19.320 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-12-15T17:00:19.322 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-12-15T17:00:19.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-15T17:00:19.409 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-15T17:00:19.440 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-15T17:00:19.506 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-15T17:00:19.506 INFO:tasks.ceph:role: osd.2 2023-12-15T17:00:19.506 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2023-12-15T17:00:19.507 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-15T17:00:19.581 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T17:00:19.581 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T17:00:19.581 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T17:00:19.581 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T17:00:19.581 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T17:00:19.581 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-12-15T17:00:19.581 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T17:00:19.581 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T17:00:19.581 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T17:00:19.581 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T17:00:20.070 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-12-15T17:00:20.072 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-12-15T17:00:20.072 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-15T17:00:20.225 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-15T17:00:20.259 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-15T17:00:20.327 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-15T17:00:20.327 INFO:tasks.ceph:role: osd.3 2023-12-15T17:00:20.327 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com 2023-12-15T17:00:20.327 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-15T17:00:20.402 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T17:00:20.403 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T17:00:20.403 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T17:00:20.403 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T17:00:20.403 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T17:00:20.403 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-12-15T17:00:20.403 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T17:00:20.403 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T17:00:20.404 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T17:00:20.404 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T17:00:20.908 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-12-15T17:00:20.910 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-12-15T17:00:20.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-15T17:00:21.044 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-15T17:00:21.079 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:21.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:21.174+0000 7f95a2010500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-15T17:00:21.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:21.174+0000 7f95a2010500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-15T17:00:21.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:21.174+0000 7f95a2010500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-15T17:00:21.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:21.174+0000 7f95a2010500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-15T17:00:21.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:21.174+0000 7f95a2010500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-15T17:00:21.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:21.175+0000 7f95a2010500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-15T17:00:24.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-15T17:00:24.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:24.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:24.143+0000 7f10a2b39500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-15T17:00:24.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:24.143+0000 7f10a2b39500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-15T17:00:24.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:24.143+0000 7f10a2b39500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-15T17:00:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:24.143+0000 7f10a2b39500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-15T17:00:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:24.143+0000 7f10a2b39500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-15T17:00:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:24.144+0000 7f10a2b39500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-15T17:00:26.978 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-15T17:00:27.007 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:27.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:27.095+0000 7fd5a83fe500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-15T17:00:27.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:27.095+0000 7fd5a83fe500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-15T17:00:27.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:27.095+0000 7fd5a83fe500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-15T17:00:27.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:27.095+0000 7fd5a83fe500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-15T17:00:27.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:27.095+0000 7fd5a83fe500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-15T17:00:27.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:27.096+0000 7fd5a83fe500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-15T17:00:29.936 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-15T17:00:29.967 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:30.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:30.056+0000 7f64450d8500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-15T17:00:30.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:30.056+0000 7f64450d8500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-15T17:00:30.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:30.056+0000 7f64450d8500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-15T17:00:30.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:30.056+0000 7f64450d8500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-15T17:00:30.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:30.056+0000 7f64450d8500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-15T17:00:30.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:30.057+0000 7f64450d8500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-15T17:00:32.895 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-15T17:00:32.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-15T17:00:32.957 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-15T17:00:32.957 INFO:tasks.ceph:role: osd.4 2023-12-15T17:00:32.958 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com 2023-12-15T17:00:32.958 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-15T17:00:33.035 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T17:00:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T17:00:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T17:00:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T17:00:33.035 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T17:00:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-12-15T17:00:33.036 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T17:00:33.036 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T17:00:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T17:00:33.036 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T17:00:33.549 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2023-12-15T17:00:33.551 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-12-15T17:00:33.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-15T17:00:33.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-15T17:00:33.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-15T17:00:33.766 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-15T17:00:33.766 INFO:tasks.ceph:role: osd.5 2023-12-15T17:00:33.766 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com 2023-12-15T17:00:33.766 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-15T17:00:33.841 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T17:00:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T17:00:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T17:00:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T17:00:33.842 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T17:00:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-12-15T17:00:33.842 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T17:00:33.842 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T17:00:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T17:00:33.842 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T17:00:34.294 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2023-12-15T17:00:34.296 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-12-15T17:00:34.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-15T17:00:34.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-15T17:00:34.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-15T17:00:34.486 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-15T17:00:34.486 INFO:tasks.ceph:role: osd.6 2023-12-15T17:00:34.486 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com 2023-12-15T17:00:34.486 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-15T17:00:34.561 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T17:00:34.561 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T17:00:34.561 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T17:00:34.561 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T17:00:34.561 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T17:00:34.561 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-12-15T17:00:34.561 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T17:00:34.561 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T17:00:34.561 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T17:00:34.561 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T17:00:35.046 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2023-12-15T17:00:35.048 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-12-15T17:00:35.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-15T17:00:35.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-15T17:00:35.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-15T17:00:35.304 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-15T17:00:35.304 INFO:tasks.ceph:role: osd.7 2023-12-15T17:00:35.304 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com 2023-12-15T17:00:35.304 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-15T17:00:35.380 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T17:00:35.380 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T17:00:35.380 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T17:00:35.380 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T17:00:35.380 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T17:00:35.381 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-12-15T17:00:35.381 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T17:00:35.381 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T17:00:35.381 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T17:00:35.381 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T17:00:35.898 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2023-12-15T17:00:35.900 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-12-15T17:00:35.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-15T17:00:36.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-15T17:00:36.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:36.159 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:36.157+0000 7f56e7812500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-15T17:00:36.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:36.158+0000 7f56e7812500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-15T17:00:36.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:36.158+0000 7f56e7812500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-15T17:00:36.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:36.158+0000 7f56e7812500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-15T17:00:36.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:36.158+0000 7f56e7812500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-15T17:00:36.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:36.158+0000 7f56e7812500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-15T17:00:39.007 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-15T17:00:39.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:39.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:39.127+0000 7ff1f4ed4500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-15T17:00:39.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:39.128+0000 7ff1f4ed4500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-15T17:00:39.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:39.128+0000 7ff1f4ed4500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-15T17:00:39.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:39.128+0000 7ff1f4ed4500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-15T17:00:39.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:39.128+0000 7ff1f4ed4500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-15T17:00:39.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:39.128+0000 7ff1f4ed4500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-15T17:00:41.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-15T17:00:42.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:42.096 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:42.094+0000 7f5b92174500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-15T17:00:42.096 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:42.095+0000 7f5b92174500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-15T17:00:42.097 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:42.095+0000 7f5b92174500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-15T17:00:42.097 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:42.095+0000 7f5b92174500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-15T17:00:42.097 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:42.095+0000 7f5b92174500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-15T17:00:42.097 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:42.095+0000 7f5b92174500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-15T17:00:44.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-15T17:00:44.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:45.056 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:45.055+0000 7fd57b09c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-15T17:00:45.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:45.055+0000 7fd57b09c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-15T17:00:45.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:45.055+0000 7fd57b09c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-15T17:00:45.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:45.055+0000 7fd57b09c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-15T17:00:45.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:45.055+0000 7fd57b09c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-15T17:00:45.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:00:45.055+0000 7fd57b09c500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-15T17:00:47.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-15T17:00:47.928 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-15T17:00:47.928 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:47.928 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-15T17:00:47.961 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:47.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-15T17:00:48.030 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:48.030 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-15T17:00:48.100 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:48.100 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-15T17:00:48.170 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:48.170 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-15T17:00:48.239 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:48.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-15T17:00:48.271 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:48.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-15T17:00:48.342 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:48.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-15T17:00:48.412 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:48.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-15T17:00:48.482 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:48.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-15T17:00:48.551 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:00:48.551 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-15T17:00:48.569 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:00:48.570 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-15T17:00:48.607 INFO:tasks.ceph:Adding keys to all mons... 2023-12-15T17:00:48.607 DEBUG:teuthology.orchestra.run.smithi069:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-15T17:00:48.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-15T17:00:48.652 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.y] 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAhhnxlRc2lLBAAGn5J0UQ7RFJQ4G0AqTyo5w== 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAlhnxl9cZtChAAH1unl2Kk8e6dKb/9L+IxTw== 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAohnxlG5qTCBAAcYFfqbuXD3o75WReZZjNDg== 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQArhnxlPmezBRAAyam6n0r3kDZDBb/rR17xrg== 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.3] 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAuhnxlJEVyAxAA7jIUTQun19/QJlhl/A9ZlQ== 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.x] 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAhhnxl11VZMBAAS1JUzvoq2VmmDkZM4Frxpw== 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.4] 2023-12-15T17:00:48.653 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQA0hnxl7Ox3CRAAGQhGoJHLHT+6FFL7Axl6aA== 2023-12-15T17:00:48.654 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.5] 2023-12-15T17:00:48.654 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQA3hnxlrECwBxAA5UsmcHf5qnTu8mrWM1l+tw== 2023-12-15T17:00:48.654 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.6] 2023-12-15T17:00:48.654 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQA6hnxls1y2BRAAdD7h6QN3ON+/TTzWSOxBjg== 2023-12-15T17:00:48.654 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.7] 2023-12-15T17:00:48.654 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQA9hnxlGaJZAxAAE4jStYG4hX44SS5AFkyTtg== 2023-12-15T17:00:48.654 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2023-12-15T17:00:48.654 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAhhnxlm0UGMxAABG+jO3GDVuzP7Geai+mOvw== 2023-12-15T17:00:48.654 INFO:teuthology.orchestra.run.smithi069.stdout:[client.1] 2023-12-15T17:00:48.654 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAhhnxlVgVuNhAAyBf0gigQPjrhuycyiBW/pQ== 2023-12-15T17:00:48.675 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.y] 2023-12-15T17:00:48.675 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAhhnxlRc2lLBAAGn5J0UQ7RFJQ4G0AqTyo5w== 2023-12-15T17:00:48.675 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.0] 2023-12-15T17:00:48.675 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAlhnxl9cZtChAAH1unl2Kk8e6dKb/9L+IxTw== 2023-12-15T17:00:48.675 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.1] 2023-12-15T17:00:48.675 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAohnxlG5qTCBAAcYFfqbuXD3o75WReZZjNDg== 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.2] 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQArhnxlPmezBRAAyam6n0r3kDZDBb/rR17xrg== 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.3] 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAuhnxlJEVyAxAA7jIUTQun19/QJlhl/A9ZlQ== 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.x] 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAhhnxl11VZMBAAS1JUzvoq2VmmDkZM4Frxpw== 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.4] 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQA0hnxl7Ox3CRAAGQhGoJHLHT+6FFL7Axl6aA== 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.5] 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQA3hnxlrECwBxAA5UsmcHf5qnTu8mrWM1l+tw== 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.6] 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQA6hnxls1y2BRAAdD7h6QN3ON+/TTzWSOxBjg== 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.7] 2023-12-15T17:00:48.676 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQA9hnxlGaJZAxAAE4jStYG4hX44SS5AFkyTtg== 2023-12-15T17:00:48.677 INFO:teuthology.orchestra.run.smithi192.stdout:[client.0] 2023-12-15T17:00:48.677 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAhhnxlm0UGMxAABG+jO3GDVuzP7Geai+mOvw== 2023-12-15T17:00:48.677 INFO:teuthology.orchestra.run.smithi192.stdout:[client.1] 2023-12-15T17:00:48.677 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAhhnxlVgVuNhAAyBf0gigQPjrhuycyiBW/pQ== 2023-12-15T17:00:48.678 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-15T17:00:48.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-15T17:00:48.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:48.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:48.827 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:48.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:48.880 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:48.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:48.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:48.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:48.986 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-15T17:00:48.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-15T17:00:49.040 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:49.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:49.093 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:49.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:49.147 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:49.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:49.200 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:49.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T17:00:49.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-15T17:00:49.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-15T17:00:49.311 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-15T17:00:49.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-15T17:00:49.368 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-15T17:00:49.368 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-15T17:00:49.395 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-15T17:00:49.491 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-15T17:00:49.517 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-15T17:00:49.585 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-15T17:00:49.680 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-15T17:00:49.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-15T17:00:49.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-15T17:00:49.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-15T17:00:49.859 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:49.861 DEBUG:teuthology.orchestra.run.smithi192:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-15T17:00:49.914 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-15T17:00:49.915 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-15T17:00:49.915 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-15T17:00:49.916 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-15T17:00:49.918 INFO:tasks.ceph.mon.a:Started 2023-12-15T17:00:49.918 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-15T17:00:49.918 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-15T17:00:49.919 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-15T17:00:49.921 INFO:tasks.ceph.mon.c:Started 2023-12-15T17:00:49.921 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-15T17:00:49.921 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-15T17:00:49.922 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-15T17:00:49.957 INFO:tasks.ceph.mon.b:Started 2023-12-15T17:00:49.957 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-15T17:00:49.957 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-15T17:00:49.957 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-12-15T17:00:49.959 INFO:tasks.ceph.mgr.y:Started 2023-12-15T17:00:49.959 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-15T17:00:49.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-12-15T17:00:49.961 INFO:tasks.ceph.mgr.x:Started 2023-12-15T17:00:49.962 INFO:tasks.ceph:Setting crush tunables to default 2023-12-15T17:00:49.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd crush tunables default 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.059+0000 7f0290dc0640 10 monclient: get_monmap_and_config 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.059+0000 7f0290dc0640 10 monclient: build_initial_monmap 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.059+0000 7f0290dc0640 10 monclient: monmap: 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:50.060603+0000 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:50.060603+0000 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:00:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:00:50.062 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:00:50.062 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:00:50.062 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:00:50.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.059+0000 7f0290dc0640 1 Processor -- start 2023-12-15T17:00:50.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.059+0000 7f0290dc0640 1 -- start start 2023-12-15T17:00:50.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.059+0000 7f0290dc0640 10 monclient: init 2023-12-15T17:00:50.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 10 monclient: _reopen_session rank -1 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f028c0a8a00 addr v1:172.21.15.69:6789/0 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b2260 0x7f028c0a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 10 monclient(hunting): picked mon.noname-b con 0x7f028c0b2260 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f028c13f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 10 monclient(hunting): picked mon.noname-c con 0x7f028c13f080 addr v2:172.21.15.192:3300/0 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028c0aa970 con 0x7f028c0a8a00 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f028c1421b0 con 0x7f028c13f080 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028c0aac50 con 0x7f028c0b2260 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 10 monclient(hunting): _renew_subs 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0290dc0640 10 monclient(hunting): authenticate will time out at 1354.376928s 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0289d9b640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b2260 msgr2=0x7f028c0a8de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-12-15T17:00:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f028a59c640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f028c0a8a00 legacy=0x7f028c1425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-12-15T17:00:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f0289d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b2260 0x7f028c0a8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-15T17:00:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f028ad9d640 1 -- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 msgr2=0x7f028c13f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.192:3300/0 2023-12-15T17:00:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.060+0000 7f028ad9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f028c13f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-15T17:00:50.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.261+0000 7f0289d9b640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b2260 msgr2=0x7f028c0a8de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-12-15T17:00:50.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.261+0000 7f028a59c640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f028c0a8a00 legacy=0x7f028c1425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-12-15T17:00:50.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.261+0000 7f0289d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b2260 0x7f028c0a8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-15T17:00:50.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.261+0000 7f028ad9d640 1 -- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 msgr2=0x7f028c13f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.192:3300/0 2023-12-15T17:00:50.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.261+0000 7f028ad9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f028c13f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-15T17:00:50.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.662+0000 7f0289d9b640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b2260 msgr2=0x7f028c0a8de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-12-15T17:00:50.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.662+0000 7f028a59c640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f028c0a8a00 legacy=0x7f028c1425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-12-15T17:00:50.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.662+0000 7f0289d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b2260 0x7f028c0a8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-15T17:00:50.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.662+0000 7f028ad9d640 1 -- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 msgr2=0x7f028c13f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.192:3300/0 2023-12-15T17:00:50.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:50.662+0000 7f028ad9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f028c13f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-15T17:00:51.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:51.463+0000 7f0289d9b640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b2260 msgr2=0x7f028c0a8de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-12-15T17:00:51.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:51.463+0000 7f028a59c640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f028c0a8a00 legacy=0x7f028c1425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-12-15T17:00:51.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:51.463+0000 7f0289d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b2260 0x7f028c0a8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-15T17:00:51.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:51.463+0000 7f028ad9d640 1 -- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 msgr2=0x7f028c13f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.192:3300/0 2023-12-15T17:00:51.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:51.463+0000 7f028ad9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f028c13f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-15T17:00:53.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 10 monclient(hunting): tick 2023-12-15T17:00:53.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 10 monclient(hunting): _check_auth_tickets 2023-12-15T17:00:53.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 1 monclient(hunting): continuing hunt 2023-12-15T17:00:53.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 10 monclient(hunting): _reopen_session rank -1 2023-12-15T17:00:53.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b2260 msgr2=0x7f028c0a8de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-15T17:00:53.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b2260 0x7f028c0a8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 1 -- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 msgr2=0x7f028c13f530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f028c13f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f028c0a8a00 legacy=0x7f028c1425d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 10 monclient: _add_conns ranks=[1,3] 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f026c0036c0 addr v1:172.21.15.192:6789/0 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f026c007410 0x7f026c0098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f026c007410 addr v2:172.21.15.69:3300/0 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028c0aaf30 con 0x7f026c0036c0 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f028c1421b0 con 0x7f026c007410 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f027bfff640 10 monclient(hunting): _renew_subs 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f0289d9b640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f026c007410 msgr2=0x7f026c0098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f0289d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f026c007410 0x7f026c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-15T17:00:53.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.060+0000 7f028a59c640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7f026c0036c0 legacy=0x7f026c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2023-12-15T17:00:53.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.261+0000 7f0289d9b640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f026c007410 msgr2=0x7f026c0098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-12-15T17:00:53.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.261+0000 7f0289d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f026c007410 0x7f026c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-15T17:00:53.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.261+0000 7f028a59c640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7f026c0036c0 legacy=0x7f026c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2023-12-15T17:00:53.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.662+0000 7f0289d9b640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f026c007410 msgr2=0x7f026c0098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-12-15T17:00:53.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.662+0000 7f0289d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f026c007410 0x7f026c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-15T17:00:53.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:53.663+0000 7f028a59c640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7f026c0036c0 legacy=0x7f026c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2023-12-15T17:00:54.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:54.463+0000 7f0289d9b640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f026c007410 msgr2=0x7f026c0098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-12-15T17:00:54.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:54.463+0000 7f0289d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f026c007410 0x7f026c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-15T17:00:54.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:54.464+0000 7f028a59c640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7f026c0036c0 legacy=0x7f026c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2023-12-15T17:00:56.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 10 monclient(hunting): tick 2023-12-15T17:00:56.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 10 monclient(hunting): _check_auth_tickets 2023-12-15T17:00:56.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 1 monclient(hunting): continuing hunt 2023-12-15T17:00:56.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 10 monclient(hunting): _reopen_session rank -1 2023-12-15T17:00:56.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f026c007410 msgr2=0x7f026c0098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-15T17:00:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f026c007410 0x7f026c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:00:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7f026c0036c0 legacy=0x7f026c005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-15T17:00:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-12-15T17:00:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f028ad9d640 1 -- reap_dead start 2023-12-15T17:00:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f026c009fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:00:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f028c13f080 addr v2:172.21.15.192:3300/0 2023-12-15T17:00:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f026c0036c0 addr v1:172.21.15.192:6789/0 2023-12-15T17:00:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f026c00a510 addr v1:172.21.15.69:6789/0 2023-12-15T17:00:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028c0aa970 con 0x7f026c00a510 2023-12-15T17:00:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028c0aaf30 con 0x7f026c0036c0 2023-12-15T17:00:56.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:00:56.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f028c1421b0 con 0x7f028c13f080 2023-12-15T17:00:56.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f027bfff640 10 monclient(hunting): _renew_subs 2023-12-15T17:00:56.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f028a59c640 1 -- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 msgr2=0x7f026c009fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.192:3300/0 2023-12-15T17:00:56.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f028ad9d640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f026c00a510 legacy=0x7f026c00d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-12-15T17:00:56.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f028a59c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f026c009fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-15T17:00:56.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.061+0000 7f0289d9b640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7f026c0036c0 legacy=0x7f026c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2023-12-15T17:00:56.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.262+0000 7f028ad9d640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f026c00a510 legacy=0x7f026c00d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-12-15T17:00:56.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.262+0000 7f028a59c640 1 -- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 msgr2=0x7f026c009fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.192:3300/0 2023-12-15T17:00:56.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.262+0000 7f028a59c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f026c009fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-15T17:00:56.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.263+0000 7f0289d9b640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7f026c0036c0 legacy=0x7f026c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2023-12-15T17:00:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.663+0000 7f028ad9d640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f026c00a510 legacy=0x7f026c00d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-12-15T17:00:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.663+0000 7f028a59c640 1 -- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 msgr2=0x7f026c009fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.192:3300/0 2023-12-15T17:00:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.663+0000 7f028a59c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f026c009fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-15T17:00:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:56.664+0000 7f0289d9b640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7f026c0036c0 legacy=0x7f026c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2023-12-15T17:00:57.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:57.464+0000 7f028ad9d640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f026c00a510 legacy=0x7f026c00d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2023-12-15T17:00:57.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:57.464+0000 7f028a59c640 1 -- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 msgr2=0x7f026c009fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.192:3300/0 2023-12-15T17:00:57.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:57.464+0000 7f028a59c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f026c009fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-15T17:00:57.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:57.465+0000 7f0289d9b640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7f026c0036c0 legacy=0x7f026c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2023-12-15T17:00:57.547 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:00:07.486 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:00:57.861 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:00:07.800 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:00:57.920 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:00:07.865 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:00:57.921 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:00:07.866 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:00:58.232 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:00:08.177 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:00:58.233 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:00:08.178 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:00:59.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 10 monclient(hunting): tick 2023-12-15T17:00:59.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 10 monclient(hunting): _check_auth_tickets 2023-12-15T17:00:59.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 1 monclient(hunting): continuing hunt 2023-12-15T17:00:59.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 10 monclient(hunting): _reopen_session rank -1 2023-12-15T17:00:59.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 1 -- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 msgr2=0x7f026c009fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-15T17:00:59.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f028c13f080 0x7f026c009fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7f026c0036c0 legacy=0x7f026c005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f026c00a510 legacy=0x7f026c00d170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 10 monclient: _add_conns ranks=[3,2] 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 0x7f026c0108a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f026c00a930 addr v2:172.21.15.69:3300/0 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 0x7f026c0131d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f026c010de0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f028c1421b0 con 0x7f026c00a930 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028c0aac50 con 0x7f026c010de0 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f027bfff640 10 monclient(hunting): _renew_subs 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f028a59c640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 msgr2=0x7f026c0108a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f0289d9b640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 msgr2=0x7f026c0131d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-12-15T17:00:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f028a59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 0x7f026c0108a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-15T17:00:59.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.062+0000 7f0289d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 0x7f026c0131d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-15T17:00:59.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.263+0000 7f028a59c640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 msgr2=0x7f026c0108a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2023-12-15T17:00:59.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.263+0000 7f0289d9b640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 msgr2=0x7f026c0131d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2023-12-15T17:00:59.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.263+0000 7f028a59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 0x7f026c0108a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-15T17:00:59.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.263+0000 7f0289d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 0x7f026c0131d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-15T17:00:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.698+0000 7f028a59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 0x7f026c0108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:00:59.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.714+0000 7f0289d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 0x7f026c0131d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:00:59.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.813+0000 7f0289d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 0x7f026c0131d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:45260/0 (socket says 172.21.15.69:45260) 2023-12-15T17:00:59.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.813+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 learned_addr learned my addr 172.21.15.69:0/2511286904 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:00:59.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.813+0000 7f0289d9b640 10 monclient(hunting): get_auth_request con 0x7f026c010de0 auth_method 0 2023-12-15T17:00:59.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.813+0000 7f0289d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:00:59.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.813+0000 7f0289d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:00:59.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.813+0000 7f0289d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:00:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.819+0000 7f028a59c640 10 monclient(hunting): get_auth_request con 0x7f026c00a930 auth_method 0 2023-12-15T17:00:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.819+0000 7f028a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:00:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.819+0000 7f028a59c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:00:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.819+0000 7f028a59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:00:59.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.843+0000 7f028a59c640 1 -- 172.21.15.69:0/2511286904 >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 msgr2=0x7f026c0108a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-15T17:00:59.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.843+0000 7f028a59c640 1 -- 172.21.15.69:0/2511286904 >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 msgr2=0x7f026c0108a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-15T17:00:59.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.843+0000 7f028a59c640 1 --2- 172.21.15.69:0/2511286904 >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 0x7f026c0108a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-15T17:00:59.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.843+0000 7f028a59c640 1 --2- 172.21.15.69:0/2511286904 >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 0x7f026c0108a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-15T17:00:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.882+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 msgr2=0x7f026c0131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-15T17:00:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.882+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 msgr2=0x7f026c0131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-15T17:00:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.882+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 0x7f026c0131d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-15T17:00:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:00:59.882+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 0x7f026c0131d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-15T17:01:00.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.645+0000 7f028a59c640 1 --2- 172.21.15.69:0/2511286904 >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 0x7f026c0108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:00.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.646+0000 7f028a59c640 10 monclient(hunting): get_auth_request con 0x7f026c00a930 auth_method 0 2023-12-15T17:01:00.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.646+0000 7f028a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:00.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.646+0000 7f028a59c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:00.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.646+0000 7f028a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:01:00.663 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.662+0000 7f028a59c640 1 -- 172.21.15.69:0/2511286904 >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 msgr2=0x7f026c0108a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-15T17:01:00.663 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.662+0000 7f028a59c640 1 -- 172.21.15.69:0/2511286904 >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 msgr2=0x7f026c0108a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-15T17:01:00.663 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.662+0000 7f028a59c640 1 --2- 172.21.15.69:0/2511286904 >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 0x7f026c0108a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-15T17:01:00.663 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.662+0000 7f028a59c640 1 --2- 172.21.15.69:0/2511286904 >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 0x7f026c0108a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-15T17:01:00.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.762+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 0x7f026c0131d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:00.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.765+0000 7f0289d9b640 10 monclient(hunting): get_auth_request con 0x7f026c010de0 auth_method 0 2023-12-15T17:01:00.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.765+0000 7f0289d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:00.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.765+0000 7f0289d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:00.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.765+0000 7f0289d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:01:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.766+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 msgr2=0x7f026c0131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-15T17:01:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.766+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 msgr2=0x7f026c0131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-15T17:01:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.766+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 0x7f026c0131d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-15T17:01:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:00.766+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 0x7f026c0131d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-15T17:01:02.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 10 monclient(hunting): tick 2023-12-15T17:01:02.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 10 monclient(hunting): _check_auth_tickets 2023-12-15T17:01:02.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 1 monclient(hunting): continuing hunt 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 10 monclient(hunting): _reopen_session rank -1 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 msgr2=0x7f026c0131d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c010de0 0x7f026c0131d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 1 -- 172.21.15.69:0/2511286904 >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 msgr2=0x7f026c0108a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 1 --2- 172.21.15.69:0/2511286904 >> v2:172.21.15.69:3300/0 conn(0x7f026c00a930 0x7f026c0108a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f028ad9d640 1 -- 172.21.15.69:0/2511286904 reap_dead start 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f026c0036c0 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f026c006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f026c009fa0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f026c006f60 addr v1:172.21.15.69:6789/0 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 1 -- 172.21.15.69:0/2511286904 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c0099d0 con 0x7f026c006f60 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 1 -- 172.21.15.69:0/2511286904 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028c0aa970 con 0x7f026c0036c0 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 1 -- 172.21.15.69:0/2511286904 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028c1421b0 con 0x7f026c009fa0 2023-12-15T17:01:02.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.062+0000 7f027bfff640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:02.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.065+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f026c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:02.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.067+0000 7f0289d9b640 10 monclient(hunting): get_auth_request con 0x7f026c009fa0 auth_method 0 2023-12-15T17:01:02.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.067+0000 7f0289d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:02.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.067+0000 7f0289d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:02.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.067+0000 7f0289d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:02.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.068+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 msgr2=0x7f026c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-15T17:01:02.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.068+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 msgr2=0x7f026c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-15T17:01:02.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.068+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f026c006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-15T17:01:02.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.068+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f026c006a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-15T17:01:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.269+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f026c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.269+0000 7f0289d9b640 10 monclient(hunting): get_auth_request con 0x7f026c009fa0 auth_method 0 2023-12-15T17:01:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.269+0000 7f0289d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.269+0000 7f0289d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.269+0000 7f0289d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:01:02.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.270+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 msgr2=0x7f026c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-15T17:01:02.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.270+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 msgr2=0x7f026c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-15T17:01:02.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.270+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f026c006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-15T17:01:02.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.270+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f026c006a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-15T17:01:02.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.681+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f026c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:02.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.681+0000 7f0289d9b640 10 monclient(hunting): get_auth_request con 0x7f026c009fa0 auth_method 0 2023-12-15T17:01:02.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.681+0000 7f0289d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:02.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.681+0000 7f0289d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:02.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.681+0000 7f0289d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:01:02.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.682+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 msgr2=0x7f026c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-15T17:01:02.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.682+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 msgr2=0x7f026c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-15T17:01:02.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.682+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f026c006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-15T17:01:02.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.682+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f026c006a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-15T17:01:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.775+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794915636 0 0) 0x7f026c0099d0 con 0x7f026c006f60 2023-12-15T17:01:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.775+0000 7f028959a640 10 monclient(hunting): _negotiate 2023-12-15T17:01:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.775+0000 7f028959a640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.775+0000 7f028959a640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.775+0000 7f028959a640 10 monclient(hunting): my global_id is 4098 2023-12-15T17:01:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.775+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0274003520 con 0x7f026c006f60 2023-12-15T17:01:02.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.801+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196133490 0 0) 0x7f028c0aa970 con 0x7f026c0036c0 2023-12-15T17:01:02.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.801+0000 7f028959a640 10 monclient(hunting): _negotiate 2023-12-15T17:01:02.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.801+0000 7f028959a640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:02.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.801+0000 7f028959a640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:02.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.801+0000 7f028959a640 10 monclient(hunting): my global_id is 4102 2023-12-15T17:01:02.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.801+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f026c0099d0 con 0x7f026c0036c0 2023-12-15T17:01:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1159317587 0 0) 0x7f0274003520 con 0x7f026c006f60 2023-12-15T17:01:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 msgr2=0x7f026c006a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-15T17:01:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f026c006a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 >> v1:172.21.15.192:6789/0 conn(0x7f026c0036c0 legacy=0x7f026c006310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient: _finish_hunting 0 2023-12-15T17:01:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 monclient: found mon.noname-a-legacy 2023-12-15T17:01:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.69:6789/0 2023-12-15T17:01:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028c0aaf30 con 0x7f026c006f60 2023-12-15T17:01:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient: _finish_auth 0 2023-12-15T17:01:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient: _check_auth_tickets 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2578675164 0 0) 0x7f0284004240 con 0x7f026c006f60 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f0290dc0640 5 monclient: authenticate success, global_id 4098 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f0290dc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient: dump: 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] but i'm connected to v1:172.21.15.69:6789/0, reconnecting 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 >> v1:172.21.15.69:6789/0 conn(0x7f026c006f60 legacy=0x7f026c004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f0274003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient(hunting): picked mon.c con 0x7f026c009fa0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 0x7f02740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient(hunting): picked mon.b con 0x7f0274004090 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0274004d00 0x7f0274005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient(hunting): picked mon.a con 0x7f0274004d00 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028c1421b0 con 0x7f0274004d00 2023-12-15T17:01:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027400bb90 con 0x7f0274004090 2023-12-15T17:01:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027400bd50 con 0x7f026c009fa0 2023-12-15T17:01:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f028959a640 10 monclient(hunting): _finish_auth 0 2023-12-15T17:01:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.854+0000 7f0290dc0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.901+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 0x7f02740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.903+0000 7f0289d9b640 10 monclient(hunting): get_auth_request con 0x7f0274004090 auth_method 0 2023-12-15T17:01:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.903+0000 7f0289d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.903+0000 7f0289d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.903+0000 7f0289d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:01:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.906+0000 7f0289d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.906+0000 7f0289d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.906+0000 7f0289d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-15T17:01:02.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.909+0000 7f028a59c640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f0274003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2023-12-15T17:01:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 msgr2=0x7f0274003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f026c009fa0 0x7f0274003b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0274004d00 msgr2=0x7f0274005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:02.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0274004d00 0x7f0274005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:02.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 10 monclient: _finish_hunting 0 2023-12-15T17:01:02.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 1 monclient: found mon.b 2023-12-15T17:01:02.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:02.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:02.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 1 -- 172.21.15.69:0/2511286904 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0274003520 con 0x7f0274004090 2023-12-15T17:01:02.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 10 monclient: _finish_auth 0 2023-12-15T17:01:02.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 10 monclient: _check_auth_tickets 2023-12-15T17:01:02.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.915+0000 7f0289d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.938+0000 7f0289d9b640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 0x7f02740047c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0280009fd0 tx=0x7f02740057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b147363b2f58a116 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:02.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028002dc30 con 0x7f0274004090 2023-12-15T17:01:02.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f028959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:02.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f028959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:02.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f028959a640 10 monclient: dump: 2023-12-15T17:01:02.974 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:02.974 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0280031070 con 0x7f0274004090 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f028959a640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f028959a640 1 -- 172.21.15.69:0/2511286904 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280037400 con 0x7f0274004090 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f028959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f028959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f028959a640 10 monclient: dump: 2023-12-15T17:01:02.975 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:02.976 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:02.976 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:02.976 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:02.976 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:02.976 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:02.976 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:02.976 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:02.976 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:02.976 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:02.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f0290dc0640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:02.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.973+0000 7f0290dc0640 4 set_mon_vals no callback set 2023-12-15T17:01:02.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.974+0000 7f0290dc0640 10 monclient: shutdown 2023-12-15T17:01:02.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.974+0000 7f0290dc0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:02.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.974+0000 7f0290dc0640 1 -- 172.21.15.69:0/2511286904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 msgr2=0x7f02740047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:02.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.974+0000 7f0290dc0640 1 --2- 172.21.15.69:0/2511286904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 0x7f02740047c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0280009fd0 tx=0x7f02740057f0 comp rx=0 tx=0).stop 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.974+0000 7f028ad9d640 1 -- 172.21.15.69:0/2511286904 reap_dead start 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.974+0000 7f0290dc0640 1 -- 172.21.15.69:0/2511286904 shutdown_connections 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.974+0000 7f0290dc0640 1 -- 172.21.15.69:0/2511286904 >> 172.21.15.69:0/2511286904 conn(0x7f028c061a90 msgr2=0x7f028c0b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.974+0000 7f0290dc0640 1 -- 172.21.15.69:0/2511286904 shutdown_connections 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.974+0000 7f0290dc0640 1 -- 172.21.15.69:0/2511286904 wait complete. 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.975+0000 7f0290dc0640 10 monclient: build_initial_monmap 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.975+0000 7f0290dc0640 10 monclient: monmap: 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:02.980 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:02.981 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:02.981 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:02.981 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:02.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.976+0000 7f0290dc0640 1 librados: starting msgr at 2023-12-15T17:01:02.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.976+0000 7f0290dc0640 1 librados: starting objecter 2023-12-15T17:01:02.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.976+0000 7f0290dc0640 1 Processor -- start 2023-12-15T17:01:02.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.976+0000 7f0290dc0640 1 -- start start 2023-12-15T17:01:02.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.976+0000 7f0290dc0640 1 librados: setting wanted keys 2023-12-15T17:01:02.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.976+0000 7f0290dc0640 1 librados: calling monclient init 2023-12-15T17:01:02.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.976+0000 7f0290dc0640 10 monclient: init 2023-12-15T17:01:02.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:02.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:01:02.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 0x7f028c0b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:02.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 10 monclient(hunting): picked mon.noname-b con 0x7f0274004090 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:02.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f028c0b8270 0x7f028c0b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:02.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 10 monclient(hunting): picked mon.noname-a con 0x7f028c0b8270 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:02.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b6150 0x7f028c1da060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:02.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 10 monclient(hunting): picked mon.noname-c con 0x7f028c0b6150 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028c13bec0 con 0x7f028c0b8270 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028c0b5450 con 0x7f0274004090 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028c0b56f0 con 0x7f028c0b6150 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f0290dc0640 10 monclient(hunting): authenticate will time out at 1367.292793s 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.977+0000 7f028a59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 0x7f028c0b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.978+0000 7f028a59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 0x7f028c0b0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:55008/0 (socket says 172.21.15.69:55008) 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.978+0000 7f028a59c640 1 -- 172.21.15.69:0/4148616763 learned_addr learned my addr 172.21.15.69:0/4148616763 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.978+0000 7f028a59c640 10 monclient(hunting): get_auth_request con 0x7f0274004090 auth_method 0 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.978+0000 7f028a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.978+0000 7f028a59c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:02.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.978+0000 7f028a59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:02.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.979+0000 7f028a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:02.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.979+0000 7f028a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:02.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.979+0000 7f028a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:02.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.988+0000 7f028ad9d640 1 --2- 172.21.15.69:0/4148616763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b6150 0x7f028c1da060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:02.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.990+0000 7f028a59c640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2023-12-15T17:01:02.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.990+0000 7f028a59c640 1 -- 172.21.15.69:0/4148616763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b6150 msgr2=0x7f028c1da060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:02.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.990+0000 7f028a59c640 1 --2- 172.21.15.69:0/4148616763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b6150 0x7f028c1da060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:02.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.990+0000 7f028a59c640 1 -- 172.21.15.69:0/4148616763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f028c0b8270 msgr2=0x7f028c0b5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:02.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.990+0000 7f028a59c640 1 --2- 172.21.15.69:0/4148616763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f028c0b8270 0x7f028c0b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:02.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.990+0000 7f028a59c640 10 monclient: _finish_hunting 0 2023-12-15T17:01:02.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.990+0000 7f028a59c640 1 monclient: found mon.noname-b 2023-12-15T17:01:02.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.990+0000 7f028a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:02.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.990+0000 7f028a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:01:02.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.990+0000 7f028a59c640 1 -- 172.21.15.69:0/4148616763 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028c0b1580 con 0x7f0274004090 2023-12-15T17:01:02.993 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.992+0000 7f028a59c640 1 --2- 172.21.15.69:0/4148616763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 0x7f028c0b0ae0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f027c0071a0 tx=0x7f027c008610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:02.993 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.992+0000 7f027affd640 10 client.?.objecter ms_handle_connect 0x7f0274004090 2023-12-15T17:01:02.993 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:02.992+0000 7f027affd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:03.006 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 1 -- 172.21.15.69:0/4148616763 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c005980 con 0x7f0274004090 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 10 monclient: dump: 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:03.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 10 monclient: _finish_auth 0 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 10 monclient: _check_auth_tickets 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 1 -- 172.21.15.69:0/4148616763 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027c011040 con 0x7f0274004090 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 1 -- 172.21.15.69:0/4148616763 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c015610 con 0x7f0274004090 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027affd640 10 monclient: dump: 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:03.008 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f0290dc0640 5 monclient: authenticate success, global_id 4108 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f027bfff640 4 set_mon_vals no callback set 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f0290dc0640 10 monclient: _renew_subs 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f0290dc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f0290dc0640 1 -- 172.21.15.69:0/4148616763 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028c0b65d0 con 0x7f0274004090 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f0290dc0640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f0290dc0640 10 monclient: _renew_subs 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f0290dc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f0290dc0640 1 -- 172.21.15.69:0/4148616763 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028c1da5a0 con 0x7f0274004090 2023-12-15T17:01:03.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.005+0000 7f0290dc0640 1 librados: init done 2023-12-15T17:01:03.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.006+0000 7f0290dc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:03.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.006+0000 7f0290dc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:03.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.006+0000 7f0290dc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:03.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.006+0000 7f0290dc0640 1 -- 172.21.15.69:0/4148616763 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f028c0b58b0 con 0x7f0274004090 2023-12-15T17:01:03.225 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.223+0000 7f027affd640 1 -- 172.21.15.69:0/4148616763 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f027c015af0 con 0x7f0274004090 2023-12-15T17:01:03.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.230+0000 7f027affd640 1 -- 172.21.15.69:0/4148616763 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f027c014390 con 0x7f0274004090 2023-12-15T17:01:03.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.230+0000 7f027affd640 10 client.4108.objecter ms_dispatch 0x7f028c000b90 osd_map(1..1 src has 1..1) v4 2023-12-15T17:01:03.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.230+0000 7f027affd640 3 client.4108.objecter handle_osd_map got epochs [1,1] > 0 2023-12-15T17:01:03.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.230+0000 7f027affd640 3 client.4108.objecter handle_osd_map decoding full epoch 1 2023-12-15T17:01:03.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.230+0000 7f027affd640 20 client.4108.objecter dump_active .. 0 homeless 2023-12-15T17:01:03.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.230+0000 7f027affd640 1 -- 172.21.15.69:0/4148616763 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f027c023060 con 0x7f0274004090 2023-12-15T17:01:03.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.230+0000 7f027affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:03.231 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.230+0000 7f027affd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:03.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.299+0000 7f0290dc0640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-15T17:01:03.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.299+0000 7f0290dc0640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-15T17:01:03.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.299+0000 7f0290dc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:03.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.299+0000 7f0290dc0640 1 -- 172.21.15.69:0/4148616763 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f028c067ac0 con 0x7f0274004090 2023-12-15T17:01:03.453 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:03.452+0000 7f70b0848040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-15T17:01:03.503 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:03.502+0000 7f70b0848040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-15T17:01:03.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.666+0000 7f027affd640 1 -- 172.21.15.69:0/4148616763 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f027c01e4d0 con 0x7f0274004090 2023-12-15T17:01:03.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.666+0000 7f027affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-15T17:01:03.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.666+0000 7f027affd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-12-15T17:01:03.668 INFO:teuthology.orchestra.run.smithi069.stderr:adjusted tunables profile to default 2023-12-15T17:01:03.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 10 librados: watch_flush enter 2023-12-15T17:01:03.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 10 librados: watch_flush exit 2023-12-15T17:01:03.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 20 client.4108.objecter shutdown clearing up homeless session... 2023-12-15T17:01:03.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 10 client.4108.objecter successfully canceled tick 2023-12-15T17:01:03.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 10 monclient: shutdown 2023-12-15T17:01:03.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:03.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 1 -- 172.21.15.69:0/4148616763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 msgr2=0x7f028c0b0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:03.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 1 --2- 172.21.15.69:0/4148616763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 0x7f028c0b0ae0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f027c0071a0 tx=0x7f027c008610 comp rx=0 tx=0).stop 2023-12-15T17:01:03.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 1 -- 172.21.15.69:0/4148616763 shutdown_connections 2023-12-15T17:01:03.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 1 --2- 172.21.15.69:0/4148616763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f028c0b6150 0x7f028c1da060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:03.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 1 --2- 172.21.15.69:0/4148616763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f028c0b8270 0x7f028c0b5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:03.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 1 --2- 172.21.15.69:0/4148616763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0274004090 0x7f028c0b0ae0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:03.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 1 -- 172.21.15.69:0/4148616763 >> 172.21.15.69:0/4148616763 conn(0x7f028c061a90 msgr2=0x7f028c073c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:03.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 1 -- 172.21.15.69:0/4148616763 shutdown_connections 2023-12-15T17:01:03.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.667+0000 7f0290dc0640 1 -- 172.21.15.69:0/4148616763 wait complete. 2023-12-15T17:01:03.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:03.668+0000 7f0290dc0640 1 librados: shutdown 2023-12-15T17:01:03.682 INFO:tasks.ceph:check_enable_crimson: False 2023-12-15T17:01:03.683 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-15T17:01:03.683 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:01:03.683 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-15T17:01:03.713 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:01:03.714 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-15T17:01:03.782 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:01:03.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-15T17:01:03.808 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:03.807+0000 7f58bb614040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-15T17:01:03.835 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:01:03.835 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-15T17:01:03.865 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:03.864+0000 7f58bb614040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-15T17:01:03.892 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:01:03.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-15T17:01:03.920 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:01:03.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-15T17:01:03.988 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:01:03.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-15T17:01:04.032 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:04.030+0000 7f70b0848040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-15T17:01:04.056 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-15T17:01:04.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-15T17:01:04.087 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:04.086+0000 7f70b0848040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-15T17:01:04.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 51cce62a-d177-4e4b-a814-7e537407f397 0 2023-12-15T17:01:04.136 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:04.132+0000 7f70b0848040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-15T17:01:04.191 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:04.189+0000 7f70b0848040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-15T17:01:04.234 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:04.233+0000 7f70b0848040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-15T17:01:04.235 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.234+0000 7f9a3fac9640 10 monclient: get_monmap_and_config 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.234+0000 7f9a3fac9640 10 monclient: build_initial_monmap 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.234+0000 7f9a3fac9640 10 monclient: monmap: 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:01:04.235610+0000 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:01:04.235610+0000 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:04.237 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 1 Processor -- start 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 1 -- start start 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 10 monclient: init 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a380b2d00 addr v1:172.21.15.69:6789/0 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a380722f0 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9a3813d3e0 0x7f9a38071470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 10 monclient(hunting): picked mon.noname-c con 0x7f9a3813d3e0 addr v2:172.21.15.192:3300/0 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380a8b00 con 0x7f9a380b2d00 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a38143900 con 0x7f9a380722f0 2023-12-15T17:01:04.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.240 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a380b30e0 con 0x7f9a3813d3e0 2023-12-15T17:01:04.240 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:04.240 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.235+0000 7f9a3fac9640 10 monclient(hunting): authenticate will time out at 1373.519235s 2023-12-15T17:01:04.240 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.236+0000 7f9a3e066640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9a3813d3e0 0x7f9a38071470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.240 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.236+0000 7f9a3d865640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9a380b2d00 0x7f9a381431f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.192:40066/0 (socket says 172.21.15.192:40066) 2023-12-15T17:01:04.240 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.236+0000 7f9a3d865640 1 -- 172.21.15.192:0/1238804718 learned_addr learned my addr 172.21.15.192:0/1238804718 (peer_addr_for_me v1:172.21.15.192:0/0) 2023-12-15T17:01:04.240 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.237+0000 7f9a3e066640 10 monclient(hunting): get_auth_request con 0x7f9a3813d3e0 auth_method 0 2023-12-15T17:01:04.240 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.237+0000 7f9a3e066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.240 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.237+0000 7f9a3e066640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.240 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.237+0000 7f9a3e066640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:04.241 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.237+0000 7f9a3d865640 1 -- 172.21.15.192:0/1238804718 >> v1:172.21.15.69:6789/0 conn(0x7f9a380b2d00 legacy=0x7f9a381431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-12-15T17:01:04.241 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.237+0000 7f9a3d865640 1 -- 172.21.15.192:0/1238804718 >> v1:172.21.15.69:6789/0 conn(0x7f9a380b2d00 legacy=0x7f9a381431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-15T17:01:04.241 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.237+0000 7f9a3d865640 1 --1- 172.21.15.192:0/1238804718 >> v1:172.21.15.69:6789/0 conn(0x7f9a380b2d00 0x7f9a381431f0 :-1 s=OPENED pgs=10 cs=1 l=1).handle_message read tag failed 2023-12-15T17:01:04.241 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.237+0000 7f9a3d865640 1 --1- 172.21.15.192:0/1238804718 >> v1:172.21.15.69:6789/0 conn(0x7f9a380b2d00 0x7f9a381431f0 :-1 s=OPENED pgs=10 cs=1 l=1).fault on lossy channel, failing 2023-12-15T17:01:04.241 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.237+0000 7f9a3c863640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.69:6789/0 2023-12-15T17:01:04.241 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.238+0000 7f9a3e066640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.241 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.238+0000 7f9a3e066640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.241 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.238+0000 7f9a3e066640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:04.242 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.241+0000 7f9a3e066640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2023-12-15T17:01:04.242 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.241+0000 7f9a3e066640 1 -- 172.21.15.192:0/1238804718 >> v1:172.21.15.192:6789/0 conn(0x7f9a380722f0 legacy=0x7f9a3813ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:04.242 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.241+0000 7f9a3e066640 1 -- 172.21.15.192:0/1238804718 >> v1:172.21.15.69:6789/0 conn(0x7f9a380b2d00 legacy=0x7f9a381431f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-12-15T17:01:04.242 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.241+0000 7f9a3e066640 10 monclient: _finish_hunting 0 2023-12-15T17:01:04.243 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.241+0000 7f9a3e066640 1 monclient: found mon.noname-c 2023-12-15T17:01:04.243 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.241+0000 7f9a3e066640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:04.243 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.241+0000 7f9a3e066640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:01:04.243 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.241+0000 7f9a3e066640 1 -- 172.21.15.192:0/1238804718 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a380a8de0 con 0x7f9a3813d3e0 2023-12-15T17:01:04.248 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.247+0000 7f9a3e066640 1 --2- 172.21.15.192:0/1238804718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a3813d3e0 0x7f9a38071470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9a38070780 tx=0x7f9a3002bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e09417aab2fa17d server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:04.254 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 1 -- 172.21.15.192:0/1238804718 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3002cc50 con 0x7f9a3813d3e0 2023-12-15T17:01:04.254 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:04.254 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:01:04.254 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 10 monclient: dump: 2023-12-15T17:01:04.254 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 10 monclient: _finish_auth 0 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 10 monclient: _check_auth_tickets 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 1 -- 172.21.15.192:0/1238804718 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3002cdb0 con 0x7f9a3813d3e0 2023-12-15T17:01:04.255 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 1 -- 172.21.15.192:0/1238804718 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30034660 con 0x7f9a3813d3e0 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3c863640 10 monclient: dump: 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:04.256 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.257 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3fac9640 5 monclient: authenticate success, global_id 4114 2023-12-15T17:01:04.257 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3fac9640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:04.257 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3fac9640 4 set_mon_vals no callback set 2023-12-15T17:01:04.260 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3fac9640 10 monclient: shutdown 2023-12-15T17:01:04.260 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3fac9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:04.260 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3fac9640 1 -- 172.21.15.192:0/1238804718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a3813d3e0 msgr2=0x7f9a38071470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:04.260 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3fac9640 1 --2- 172.21.15.192:0/1238804718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a3813d3e0 0x7f9a38071470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9a38070780 tx=0x7f9a3002bf20 comp rx=0 tx=0).stop 2023-12-15T17:01:04.260 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3fac9640 1 -- 172.21.15.192:0/1238804718 shutdown_connections 2023-12-15T17:01:04.260 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3fac9640 1 --2- 172.21.15.192:0/1238804718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a3813d3e0 0x7f9a38071470 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.260 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.253+0000 7f9a3fac9640 1 -- 172.21.15.192:0/1238804718 >> 172.21.15.192:0/1238804718 conn(0x7f9a380b5eb0 msgr2=0x7f9a380b2380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:04.260 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.254+0000 7f9a3fac9640 1 -- 172.21.15.192:0/1238804718 shutdown_connections 2023-12-15T17:01:04.260 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.254+0000 7f9a3fac9640 1 -- 172.21.15.192:0/1238804718 wait complete. 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.254+0000 7f9a3fac9640 10 monclient: build_initial_monmap 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.254+0000 7f9a3fac9640 10 monclient: monmap: 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 librados: starting msgr at 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 librados: starting objecter 2023-12-15T17:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 Processor -- start 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 -- start start 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 librados: setting wanted keys 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 librados: calling monclient init 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 10 monclient: init 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 0x7f9a3813fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 10 monclient(hunting): picked mon.noname-b con 0x7f9a380722f0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b2d00 0x7f9a3813a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 10 monclient(hunting): picked mon.noname-a con 0x7f9a380b2d00 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a3813d3e0 0x7f9a380a9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 10 monclient(hunting): picked mon.noname-c con 0x7f9a3813d3e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3813a7f0 con 0x7f9a380b2d00 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.262 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a38139e20 con 0x7f9a380722f0 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3813b5e0 con 0x7f9a3813d3e0 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.255+0000 7f9a3fac9640 10 monclient(hunting): authenticate will time out at 1373.539181s 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d865640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 0x7f9a3813fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d064640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b2d00 0x7f9a3813a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3e066640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a3813d3e0 0x7f9a380a9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d865640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 0x7f9a3813fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52754/0 (socket says 172.21.15.192:52754) 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d865640 1 -- 172.21.15.192:0/3751734350 learned_addr learned my addr 172.21.15.192:0/3751734350 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d865640 10 monclient(hunting): get_auth_request con 0x7f9a380722f0 auth_method 0 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d865640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d865640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d064640 10 monclient(hunting): get_auth_request con 0x7f9a380b2d00 auth_method 0 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d064640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3d064640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3e066640 10 monclient(hunting): get_auth_request con 0x7f9a3813d3e0 auth_method 0 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3e066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3e066640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.256+0000 7f9a3e066640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.257+0000 7f9a3d865640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.257+0000 7f9a3d865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.257+0000 7f9a3d865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.257+0000 7f9a3e066640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.257+0000 7f9a3e066640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.257+0000 7f9a3e066640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.257+0000 7f9a3d064640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.257+0000 7f9a3d064640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.257+0000 7f9a3d064640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:04.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.259+0000 7f9a3d865640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2023-12-15T17:01:04.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.259+0000 7f9a3d865640 1 -- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a3813d3e0 msgr2=0x7f9a380a9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:04.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.259+0000 7f9a3d865640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a3813d3e0 0x7f9a380a9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.259+0000 7f9a3d865640 1 -- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b2d00 msgr2=0x7f9a3813a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:04.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.259+0000 7f9a3d865640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b2d00 0x7f9a3813a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.266 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.259+0000 7f9a3d865640 10 monclient: _finish_hunting 0 2023-12-15T17:01:04.266 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.259+0000 7f9a3d865640 1 monclient: found mon.noname-b 2023-12-15T17:01:04.266 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.259+0000 7f9a3d865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:04.266 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.259+0000 7f9a3d865640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:01:04.266 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.259+0000 7f9a3d865640 1 -- 172.21.15.192:0/3751734350 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a30009a70 con 0x7f9a380722f0 2023-12-15T17:01:04.266 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.262+0000 7f9a3d865640 1 -- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 msgr2=0x7f9a3813fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-12-15T17:01:04.266 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.262+0000 7f9a3d865640 1 -- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 msgr2=0x7f9a3813fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-15T17:01:04.266 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.262+0000 7f9a3d865640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 0x7f9a3813fdf0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f9a3400c580 tx=0x7f9a3400ca40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-15T17:01:04.266 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.262+0000 7f9a3d865640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 0x7f9a3813fdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-15T17:01:04.341 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:04.339+0000 7f70b0848040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-15T17:01:04.408 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:04.407+0000 7f70b0848040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-15T17:01:04.423 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:04.422+0000 7f58bb614040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-15T17:01:04.453 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:04.452+0000 7f70b0848040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-15T17:01:04.467 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.463+0000 7f9a3d865640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 0x7f9a3813fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.467 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a3d865640 10 monclient: get_auth_request con 0x7f9a380722f0 auth_method 0 2023-12-15T17:01:04.467 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a3d865640 0 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 0x7f9a3813fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a3d865640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 0x7f9a3813fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 1 -- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 msgr2=0x7f9a3813fdf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a380722f0 0x7f9a3813fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a1c0034d0 0x7f9a1c005920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f9a1c0034d0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a3813d3e0 0x7f9a1c0084e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f9a3813d3e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b2d00 0x7f9a1c00afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f9a380b2d00 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 1 -- 172.21.15.192:0/3751734350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c00b9c0 con 0x7f9a380b2d00 2023-12-15T17:01:04.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 1 -- 172.21.15.192:0/3751734350 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c00bb00 con 0x7f9a1c0034d0 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 1 -- 172.21.15.192:0/3751734350 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c00bcc0 con 0x7f9a3813d3e0 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a2e7fc640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a3d064640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a3813d3e0 0x7f9a1c0084e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.464+0000 7f9a3e066640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b2d00 0x7f9a1c00afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3d064640 10 monclient(hunting): get_auth_request con 0x7f9a3813d3e0 auth_method 0 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3d064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3d064640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3d064640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3e066640 10 monclient(hunting): get_auth_request con 0x7f9a380b2d00 auth_method 0 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3e066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3e066640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3e066640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:01:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3d865640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a1c0034d0 0x7f9a1c005920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3d865640 10 monclient(hunting): get_auth_request con 0x7f9a1c0034d0 auth_method 0 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3d865640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.465+0000 7f9a3d865640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.466+0000 7f9a3d064640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.466+0000 7f9a3d064640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.466+0000 7f9a3d064640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.466+0000 7f9a3e066640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.466+0000 7f9a3e066640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.466+0000 7f9a3e066640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.466+0000 7f9a3d865640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.466+0000 7f9a3d865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.470 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.466+0000 7f9a3d865640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-15T17:01:04.471 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.468+0000 7f9a3d865640 10 monclient(hunting): handle_auth_done global_id 4120 payload 293 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.468+0000 7f9a3d865640 1 -- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a3813d3e0 msgr2=0x7f9a1c0084e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.468+0000 7f9a3d865640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a3813d3e0 0x7f9a1c0084e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.468+0000 7f9a3d865640 1 -- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b2d00 msgr2=0x7f9a1c00afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.468+0000 7f9a3d865640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b2d00 0x7f9a1c00afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.468+0000 7f9a3d865640 10 monclient: _finish_hunting 0 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.468+0000 7f9a3d865640 1 monclient: found mon.noname-b 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.468+0000 7f9a3d865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.468+0000 7f9a3d865640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.468+0000 7f9a3d865640 1 -- 172.21.15.192:0/3751734350 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c00bf40 con 0x7f9a1c0034d0 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.470+0000 7f9a3d865640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a1c0034d0 0x7f9a1c005920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9a3400c580 tx=0x7f9a34001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.470+0000 7f9a2e7fc640 10 client.?.objecter ms_handle_connect 0x7f9a1c0034d0 2023-12-15T17:01:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.470+0000 7f9a2e7fc640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:04.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 1 -- 172.21.15.192:0/3751734350 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a340072f0 con 0x7f9a1c0034d0 2023-12-15T17:01:04.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:04.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:04.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 10 monclient: dump: 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 10 monclient: _finish_auth 0 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 10 monclient: _check_auth_tickets 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 1 -- 172.21.15.192:0/3751734350 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a34007450 con 0x7f9a1c0034d0 2023-12-15T17:01:04.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a3fac9640 5 monclient: authenticate success, global_id 4120 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a3fac9640 10 monclient: _renew_subs 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a3fac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a3fac9640 1 -- 172.21.15.192:0/3751734350 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3813b8c0 con 0x7f9a1c0034d0 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 1 -- 172.21.15.192:0/3751734350 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a34005bd0 con 0x7f9a1c0034d0 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2e7fc640 10 monclient: dump: 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.477 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a2ffff640 4 set_mon_vals no callback set 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a3fac9640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a3fac9640 10 monclient: _renew_subs 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a3fac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a3fac9640 1 -- 172.21.15.192:0/3751734350 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a38139720 con 0x7f9a1c0034d0 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.473+0000 7f9a3fac9640 1 librados: init done 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.474+0000 7f9a3fac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.474+0000 7f9a3fac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.474+0000 7f9a3fac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:04.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.474+0000 7f9a3fac9640 1 -- 172.21.15.192:0/3751734350 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a38067ac0 con 0x7f9a1c0034d0 2023-12-15T17:01:04.479 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:04.475+0000 7f58bb614040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-15T17:01:04.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.476+0000 7f9a2e7fc640 1 -- 172.21.15.192:0/3751734350 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f9a3401b070 con 0x7f9a1c0034d0 2023-12-15T17:01:04.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.476+0000 7f9a2e7fc640 1 -- 172.21.15.192:0/3751734350 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9a34015780 con 0x7f9a1c0034d0 2023-12-15T17:01:04.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.476+0000 7f9a2e7fc640 10 client.4120.objecter ms_dispatch 0x7f9a38000b90 osd_map(2..2 src has 1..2) v4 2023-12-15T17:01:04.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.476+0000 7f9a2e7fc640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2023-12-15T17:01:04.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.476+0000 7f9a2e7fc640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2023-12-15T17:01:04.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.476+0000 7f9a2e7fc640 20 client.4120.objecter dump_active .. 0 homeless 2023-12-15T17:01:04.508 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:04.506+0000 7f70b0848040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-15T17:01:04.522 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:04.521+0000 7f58bb614040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-15T17:01:04.556 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:04.554+0000 7f70b0848040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-15T17:01:04.576 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.574+0000 7f9a2e7fc640 1 -- 172.21.15.192:0/3751734350 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f9a34005d50 con 0x7f9a1c0034d0 2023-12-15T17:01:04.576 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.574+0000 7f9a2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:04.576 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.574+0000 7f9a2e7fc640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:04.577 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:04.576+0000 7f58bb614040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-15T17:01:04.632 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:04.631+0000 7f58bb614040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-15T17:01:04.651 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.649+0000 7f9a3fac9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "51cce62a-d177-4e4b-a814-7e537407f397", "id": 0}] 2023-12-15T17:01:04.651 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.649+0000 7f9a3fac9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "51cce62a-d177-4e4b-a814-7e537407f397", "id": 0}] 2023-12-15T17:01:04.651 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.649+0000 7f9a3fac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:04.651 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.649+0000 7f9a3fac9640 1 -- 172.21.15.192:0/3751734350 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "51cce62a-d177-4e4b-a814-7e537407f397", "id": 0} v 0) v1 -- 0x7f9a38075b60 con 0x7f9a1c0034d0 2023-12-15T17:01:04.736 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:04.735+0000 7f58bb614040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-15T17:01:04.799 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.797+0000 7f9a2e7fc640 1 -- 172.21.15.192:0/3751734350 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "51cce62a-d177-4e4b-a814-7e537407f397", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9a34015990 con 0x7f9a1c0034d0 2023-12-15T17:01:04.799 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.798+0000 7f9a2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "51cce62a-d177-4e4b-a814-7e537407f397", "id": 0}] 2023-12-15T17:01:04.799 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.798+0000 7f9a2e7fc640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:04.800 INFO:teuthology.orchestra.run.smithi192.stdout:0 2023-12-15T17:01:04.801 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 10 librados: watch_flush enter 2023-12-15T17:01:04.801 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 10 librados: watch_flush exit 2023-12-15T17:01:04.801 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 20 client.4120.objecter shutdown clearing up homeless session... 2023-12-15T17:01:04.801 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 10 client.4120.objecter successfully canceled tick 2023-12-15T17:01:04.801 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 10 monclient: shutdown 2023-12-15T17:01:04.801 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:04.801 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 1 -- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a1c0034d0 msgr2=0x7f9a1c005920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:04.801 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a1c0034d0 0x7f9a1c005920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9a3400c580 tx=0x7f9a34001120 comp rx=0 tx=0).stop 2023-12-15T17:01:04.802 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 1 -- 172.21.15.192:0/3751734350 shutdown_connections 2023-12-15T17:01:04.802 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b2d00 0x7f9a1c00afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.802 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a3813d3e0 0x7f9a1c0084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.802 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 1 --2- 172.21.15.192:0/3751734350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a1c0034d0 0x7f9a1c005920 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.802 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 1 -- 172.21.15.192:0/3751734350 >> 172.21.15.192:0/3751734350 conn(0x7f9a380b5eb0 msgr2=0x7f9a380b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:04.802 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 1 -- 172.21.15.192:0/3751734350 shutdown_connections 2023-12-15T17:01:04.802 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 1 -- 172.21.15.192:0/3751734350 wait complete. 2023-12-15T17:01:04.802 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.799+0000 7f9a3fac9640 1 librados: shutdown 2023-12-15T17:01:04.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 1556db3a-b386-444f-8327-3dce2ad46d33 1 2023-12-15T17:01:04.829 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:04.828+0000 7f58bb614040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-15T17:01:04.880 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:04.879+0000 7f58bb614040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-15T17:01:04.912 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.911+0000 7f8d2b021640 10 monclient: get_monmap_and_config 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.911+0000 7f8d2b021640 10 monclient: build_initial_monmap 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.911+0000 7f8d2b021640 10 monclient: monmap: 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:01:04.912262+0000 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:01:04.912262+0000 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:04.913 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.911+0000 7f8d2b021640 1 Processor -- start 2023-12-15T17:01:04.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.911+0000 7f8d2b021640 1 -- start start 2023-12-15T17:01:04.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.911+0000 7f8d2b021640 10 monclient: init 2023-12-15T17:01:04.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:04.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-12-15T17:01:04.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 0x7f8d24142540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 10 monclient(hunting): picked mon.noname-b con 0x7f8d2406a500 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:04.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d2406a8e0 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:04.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d24142a80 0x7f8d2414c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 10 monclient(hunting): picked mon.noname-a con 0x7f8d24142a80 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:04.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d240accc0 con 0x7f8d2406a8e0 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d240726a0 con 0x7f8d24142a80 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d240727e0 con 0x7f8d2406a500 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.912+0000 7f8d2b021640 10 monclient(hunting): authenticate will time out at 1374.195901s 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d28dbd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 0x7f8d24142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d23fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f8d2406a8e0 0x7f8d24071f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:41552/0 (socket says 172.21.15.192:41552) 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d23fff640 1 -- 172.21.15.192:0/949361547 learned_addr learned my addr 172.21.15.192:0/949361547 (peer_addr_for_me v1:172.21.15.192:0/0) 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d295be640 1 --2- 172.21.15.192:0/949361547 >> v2:172.21.15.69:3300/0 conn(0x7f8d24142a80 0x7f8d2414c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d28dbd640 10 monclient(hunting): get_auth_request con 0x7f8d2406a500 auth_method 0 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d28dbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d28dbd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.917 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d28dbd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:04.918 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d295be640 10 monclient(hunting): get_auth_request con 0x7f8d24142a80 auth_method 0 2023-12-15T17:01:04.918 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d295be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.918 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d295be640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.918 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d295be640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:04.918 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d28dbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.918 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d28dbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.918 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.913+0000 7f8d28dbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:04.918 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.914+0000 7f8d295be640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.918 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.914+0000 7f8d295be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.918 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.914+0000 7f8d295be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:04.919 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.915+0000 7f8d237fe640 1 -- 172.21.15.192:0/949361547 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62420924 0 0) 0x7f8d240accc0 con 0x7f8d2406a8e0 2023-12-15T17:01:04.919 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.915+0000 7f8d28dbd640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2023-12-15T17:01:04.919 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.915+0000 7f8d28dbd640 1 -- 172.21.15.192:0/949361547 >> v2:172.21.15.69:3300/0 conn(0x7f8d24142a80 msgr2=0x7f8d2414c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:04.919 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.915+0000 7f8d28dbd640 1 --2- 172.21.15.192:0/949361547 >> v2:172.21.15.69:3300/0 conn(0x7f8d24142a80 0x7f8d2414c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.919 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.915+0000 7f8d28dbd640 1 -- 172.21.15.192:0/949361547 >> v1:172.21.15.192:6789/0 conn(0x7f8d2406a8e0 legacy=0x7f8d24071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:04.919 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.915+0000 7f8d28dbd640 10 monclient: _finish_hunting 0 2023-12-15T17:01:04.919 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.915+0000 7f8d28dbd640 1 monclient: found mon.noname-b 2023-12-15T17:01:04.919 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.915+0000 7f8d28dbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:04.919 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.915+0000 7f8d28dbd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:01:04.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.915+0000 7f8d28dbd640 1 -- 172.21.15.192:0/949361547 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d240ac8b0 con 0x7f8d2406a500 2023-12-15T17:01:04.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.915+0000 7f8d237fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-12-15T17:01:04.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.917+0000 7f8d28dbd640 1 --2- 172.21.15.192:0/949361547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 0x7f8d24142540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8d180097a0 tx=0x7f8d1802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b58e573451e86b8 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:04.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 1 -- 172.21.15.192:0/949361547 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1802e070 con 0x7f8d2406a500 2023-12-15T17:01:04.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:04.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 10 monclient: dump: 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 10 monclient: _finish_auth 0 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 10 monclient: _check_auth_tickets 2023-12-15T17:01:04.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 1 -- 172.21.15.192:0/949361547 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d180315a0 con 0x7f8d2406a500 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 1 -- 172.21.15.192:0/949361547 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18031930 con 0x7f8d2406a500 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 5 monclient: authenticate success, global_id 4106 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 4 set_mon_vals no callback set 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d237fe640 10 monclient: dump: 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.925 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:04.926 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.926 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:04.926 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:04.926 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:04.926 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 10 monclient: shutdown 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 1 -- 172.21.15.192:0/949361547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 msgr2=0x7f8d24142540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 1 --2- 172.21.15.192:0/949361547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 0x7f8d24142540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8d180097a0 tx=0x7f8d1802b840 comp rx=0 tx=0).stop 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 1 -- 172.21.15.192:0/949361547 shutdown_connections 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 1 --2- 172.21.15.192:0/949361547 >> v2:172.21.15.69:3300/0 conn(0x7f8d24142a80 0x7f8d2414c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 1 --2- 172.21.15.192:0/949361547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 0x7f8d24142540 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 1 -- 172.21.15.192:0/949361547 >> 172.21.15.192:0/949361547 conn(0x7f8d240b4660 msgr2=0x7f8d240b4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 1 -- 172.21.15.192:0/949361547 shutdown_connections 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.922+0000 7f8d2b021640 1 -- 172.21.15.192:0/949361547 wait complete. 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.923+0000 7f8d2b021640 10 monclient: build_initial_monmap 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.923+0000 7f8d2b021640 10 monclient: monmap: 2023-12-15T17:01:04.930 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.923+0000 7f8d2b021640 1 librados: starting msgr at 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.923+0000 7f8d2b021640 1 librados: starting objecter 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.923+0000 7f8d2b021640 1 Processor -- start 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 1 -- start start 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 1 librados: setting wanted keys 2023-12-15T17:01:04.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 1 librados: calling monclient init 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 10 monclient: init 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 0x7f8d2411d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 10 monclient(hunting): picked mon.noname-c con 0x7f8d2406a500 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d24142a80 0x7f8d2413c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 10 monclient(hunting): picked mon.noname-a con 0x7f8d24142a80 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d2411e270 0x7f8d2413cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 10 monclient(hunting): picked mon.noname-b con 0x7f8d2411e270 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2411f940 con 0x7f8d24142a80 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2411e750 con 0x7f8d2411e270 2023-12-15T17:01:04.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2411c610 con 0x7f8d2406a500 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.924+0000 7f8d2b021640 10 monclient(hunting): authenticate will time out at 1374.208042s 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d28dbd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 0x7f8d2411d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d23fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d24142a80 0x7f8d2413c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d295be640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d2411e270 0x7f8d2413cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d28dbd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 0x7f8d2411d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.192:45242/0 (socket says 172.21.15.192:45242) 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d28dbd640 1 -- 172.21.15.192:0/191769298 learned_addr learned my addr 172.21.15.192:0/191769298 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d28dbd640 10 monclient(hunting): get_auth_request con 0x7f8d2406a500 auth_method 0 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d28dbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d28dbd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d28dbd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d295be640 10 monclient(hunting): get_auth_request con 0x7f8d2411e270 auth_method 0 2023-12-15T17:01:04.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d295be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d295be640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d295be640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d23fff640 10 monclient(hunting): get_auth_request con 0x7f8d24142a80 auth_method 0 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d23fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d23fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d28dbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d28dbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.925+0000 7f8d28dbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.926+0000 7f8d295be640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.926+0000 7f8d295be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.926+0000 7f8d295be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.926+0000 7f8d23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.926+0000 7f8d23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.926+0000 7f8d23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:04.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.928+0000 7f8d28dbd640 10 monclient(hunting): handle_auth_done global_id 4109 payload 824 2023-12-15T17:01:04.935 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.928+0000 7f8d28dbd640 1 -- 172.21.15.192:0/191769298 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d2411e270 msgr2=0x7f8d2413cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:04.935 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.928+0000 7f8d28dbd640 1 --2- 172.21.15.192:0/191769298 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d2411e270 0x7f8d2413cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.935 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.928+0000 7f8d28dbd640 1 -- 172.21.15.192:0/191769298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d24142a80 msgr2=0x7f8d2413c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:04.935 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.928+0000 7f8d28dbd640 1 --2- 172.21.15.192:0/191769298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d24142a80 0x7f8d2413c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:04.935 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.928+0000 7f8d28dbd640 10 monclient: _finish_hunting 0 2023-12-15T17:01:04.935 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.928+0000 7f8d28dbd640 1 monclient: found mon.noname-c 2023-12-15T17:01:04.935 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.928+0000 7f8d28dbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:04.935 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.928+0000 7f8d28dbd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:01:04.935 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.928+0000 7f8d28dbd640 1 -- 172.21.15.192:0/191769298 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d18009400 con 0x7f8d2406a500 2023-12-15T17:01:04.936 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.930+0000 7f8d28dbd640 1 --2- 172.21.15.192:0/191769298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 0x7f8d2411d450 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8d18034040 tx=0x7f8d180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:04.936 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.930+0000 7f8d217fa640 10 client.?.objecter ms_handle_connect 0x7f8d2406a500 2023-12-15T17:01:04.936 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.930+0000 7f8d217fa640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:04.937 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 1 -- 172.21.15.192:0/191769298 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18032840 con 0x7f8d2406a500 2023-12-15T17:01:04.937 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:04.937 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:01:04.937 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 10 monclient: dump: 2023-12-15T17:01:04.937 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:04.937 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:04.937 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.937 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.937 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:04.937 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 10 monclient: _finish_auth 0 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 10 monclient: _check_auth_tickets 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 1 -- 172.21.15.192:0/191769298 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d180329c0 con 0x7f8d2406a500 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 1 -- 172.21.15.192:0/191769298 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18032d70 con 0x7f8d2406a500 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d2b021640 5 monclient: authenticate success, global_id 4109 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d2b021640 10 monclient: _renew_subs 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d2b021640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d2b021640 1 -- 172.21.15.192:0/191769298 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2411d990 con 0x7f8d2406a500 2023-12-15T17:01:04.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d227fc640 4 set_mon_vals no callback set 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d217fa640 10 monclient: dump: 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d2b021640 10 client.4109.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d2b021640 10 monclient: _renew_subs 2023-12-15T17:01:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d2b021640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:04.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d2b021640 1 -- 172.21.15.192:0/191769298 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d241d6fa0 con 0x7f8d2406a500 2023-12-15T17:01:04.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.935+0000 7f8d2b021640 1 librados: init done 2023-12-15T17:01:04.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.936+0000 7f8d2b021640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:04.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.936+0000 7f8d2b021640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:04.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.936+0000 7f8d2b021640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:04.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.936+0000 7f8d2b021640 1 -- 172.21.15.192:0/191769298 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d241d6b00 con 0x7f8d2406a500 2023-12-15T17:01:04.944 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:04.943+0000 7f58bb614040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-15T17:01:04.953 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:04.952+0000 7f70b0848040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-15T17:01:04.961 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.959+0000 7f8d217fa640 1 -- 172.21.15.192:0/191769298 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f8d18005d10 con 0x7f8d2406a500 2023-12-15T17:01:04.961 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.959+0000 7f8d217fa640 1 -- 172.21.15.192:0/191769298 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f8d1804e820 con 0x7f8d2406a500 2023-12-15T17:01:04.961 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.959+0000 7f8d217fa640 10 client.4109.objecter ms_dispatch 0x7f8d24000b90 osd_map(3..3 src has 1..3) v4 2023-12-15T17:01:04.961 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.959+0000 7f8d217fa640 3 client.4109.objecter handle_osd_map got epochs [3,3] > 0 2023-12-15T17:01:04.961 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.959+0000 7f8d217fa640 3 client.4109.objecter handle_osd_map decoding full epoch 3 2023-12-15T17:01:04.962 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:04.959+0000 7f8d217fa640 20 client.4109.objecter dump_active .. 0 homeless 2023-12-15T17:01:04.993 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:04.991+0000 7f58bb614040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-15T17:01:05.086 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:05.084+0000 7f70b0848040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-15T17:01:05.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.129+0000 7f8d217fa640 1 -- 172.21.15.192:0/191769298 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f8d241d6b00 con 0x7f8d2406a500 2023-12-15T17:01:05.131 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.129+0000 7f8d217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:05.131 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.129+0000 7f8d217fa640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:05.201 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.199+0000 7f8d2b021640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1556db3a-b386-444f-8327-3dce2ad46d33", "id": 1}] 2023-12-15T17:01:05.201 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.199+0000 7f8d2b021640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1556db3a-b386-444f-8327-3dce2ad46d33", "id": 1}] 2023-12-15T17:01:05.201 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.199+0000 7f8d2b021640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:05.201 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.199+0000 7f8d2b021640 1 -- 172.21.15.192:0/191769298 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1556db3a-b386-444f-8327-3dce2ad46d33", "id": 1} v 0) v1 -- 0x7f8d24075b60 con 0x7f8d2406a500 2023-12-15T17:01:05.207 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:05.206+0000 7f70b0848040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-15T17:01:05.253 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:05.252+0000 7f70b0848040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-15T17:01:05.353 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:05.351+0000 7f70b0848040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-15T17:01:05.357 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.356+0000 7f8d217fa640 1 -- 172.21.15.192:0/191769298 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1556db3a-b386-444f-8327-3dce2ad46d33", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8d180530b0 con 0x7f8d2406a500 2023-12-15T17:01:05.357 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.356+0000 7f8d217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1556db3a-b386-444f-8327-3dce2ad46d33", "id": 1}] 2023-12-15T17:01:05.357 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.356+0000 7f8d217fa640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:05.358 INFO:teuthology.orchestra.run.smithi192.stdout:1 2023-12-15T17:01:05.359 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 10 librados: watch_flush enter 2023-12-15T17:01:05.359 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 10 librados: watch_flush exit 2023-12-15T17:01:05.359 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 20 client.4109.objecter shutdown clearing up homeless session... 2023-12-15T17:01:05.359 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 10 client.4109.objecter successfully canceled tick 2023-12-15T17:01:05.359 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 10 monclient: shutdown 2023-12-15T17:01:05.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:05.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 1 -- 172.21.15.192:0/191769298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 msgr2=0x7f8d2411d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:05.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 1 --2- 172.21.15.192:0/191769298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 0x7f8d2411d450 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8d18034040 tx=0x7f8d180046c0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 1 -- 172.21.15.192:0/191769298 shutdown_connections 2023-12-15T17:01:05.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 1 --2- 172.21.15.192:0/191769298 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d2411e270 0x7f8d2413cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 1 --2- 172.21.15.192:0/191769298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d24142a80 0x7f8d2413c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 1 --2- 172.21.15.192:0/191769298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2406a500 0x7f8d2411d450 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 1 -- 172.21.15.192:0/191769298 >> 172.21.15.192:0/191769298 conn(0x7f8d240b4660 msgr2=0x7f8d24071f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:05.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 1 -- 172.21.15.192:0/191769298 shutdown_connections 2023-12-15T17:01:05.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 1 -- 172.21.15.192:0/191769298 wait complete. 2023-12-15T17:01:05.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.357+0000 7f8d2b021640 1 librados: shutdown 2023-12-15T17:01:05.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new eaac40b5-9fc0-4e13-a322-763a9f3b685f 2 2023-12-15T17:01:05.391 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:05.390+0000 7f58bb614040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-15T17:01:05.452 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:05.451+0000 7f70b0848040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-15T17:01:05.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.467+0000 7f797a24f640 10 monclient: get_monmap_and_config 2023-12-15T17:01:05.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.467+0000 7f797a24f640 10 monclient: build_initial_monmap 2023-12-15T17:01:05.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.467+0000 7f797a24f640 10 monclient: monmap: 2023-12-15T17:01:05.469 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:05.469 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:05.469 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:01:05.468597+0000 2023-12-15T17:01:05.469 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:01:05.468597+0000 2023-12-15T17:01:05.469 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:05.469 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.469 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:05.470 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:05.470 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:05.470 INFO:teuthology.orchestra.run.smithi192.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:05.470 INFO:teuthology.orchestra.run.smithi192.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:05.470 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 1 Processor -- start 2023-12-15T17:01:05.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 1 -- start start 2023-12-15T17:01:05.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 10 monclient: init 2023-12-15T17:01:05.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:05.472 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79740a8a00 0x7f79740a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 10 monclient(hunting): picked mon.noname-a con 0x7f79740a8a00 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f79740b2260 0x7f79741425d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 10 monclient(hunting): picked mon.noname-c con 0x7f79740b2260 addr v2:172.21.15.192:3300/0 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f797413f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 10 monclient(hunting): picked mon.noname-b con 0x7f7974142b10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.468+0000 7f797a24f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79741421b0 con 0x7f79740a8a00 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f797a24f640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f797a24f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79740aa970 con 0x7f79740b2260 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f797a24f640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f797a24f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79740aac50 con 0x7f7974142b10 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f797a24f640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f797a24f640 10 monclient(hunting): authenticate will time out at 1374.752356s 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f7972ffd640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f79740b2260 0x7f79741425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f7973fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f797413f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f79737fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79740a8a00 0x7f79740a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f7973fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f797413f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.192:45258/0 (socket says 172.21.15.192:45258) 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f7973fff640 1 -- 172.21.15.192:0/1049689983 learned_addr learned my addr 172.21.15.192:0/1049689983 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f7973fff640 10 monclient(hunting): get_auth_request con 0x7f7974142b10 auth_method 0 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f7973fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f7973fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.469+0000 7f7973fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f7972ffd640 10 monclient(hunting): get_auth_request con 0x7f79740b2260 auth_method 0 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f7972ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f7972ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f7972ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f79737fe640 10 monclient(hunting): get_auth_request con 0x7f79740a8a00 auth_method 0 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f79737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f79737fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f79737fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f7973fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f7973fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f7973fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f7972ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f7972ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f7972ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f79737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f79737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:05.475 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.470+0000 7f79737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:05.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.472+0000 7f7973fff640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2023-12-15T17:01:05.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.472+0000 7f7973fff640 1 -- 172.21.15.192:0/1049689983 >> v2:172.21.15.192:3300/0 conn(0x7f79740b2260 msgr2=0x7f79741425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:05.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.472+0000 7f7973fff640 1 --2- 172.21.15.192:0/1049689983 >> v2:172.21.15.192:3300/0 conn(0x7f79740b2260 0x7f79741425d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.472+0000 7f7973fff640 1 -- 172.21.15.192:0/1049689983 >> v2:172.21.15.69:3300/0 conn(0x7f79740a8a00 msgr2=0x7f79740a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:05.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.472+0000 7f7973fff640 1 --2- 172.21.15.192:0/1049689983 >> v2:172.21.15.69:3300/0 conn(0x7f79740a8a00 0x7f79740a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.472+0000 7f7973fff640 10 monclient: _finish_hunting 0 2023-12-15T17:01:05.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.472+0000 7f7973fff640 1 monclient: found mon.noname-b 2023-12-15T17:01:05.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.472+0000 7f7973fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:05.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.472+0000 7f7973fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:01:05.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.472+0000 7f7973fff640 1 -- 172.21.15.192:0/1049689983 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79740aaf30 con 0x7f7974142b10 2023-12-15T17:01:05.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.474+0000 7f7973fff640 1 --2- 172.21.15.192:0/1049689983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f797413f080 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f79740b5270 tx=0x7f796c02bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27be36b8d2fce6e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 1 -- 172.21.15.192:0/1049689983 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c02c950 con 0x7f7974142b10 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 10 monclient: dump: 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:05.479 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 10 monclient: _finish_auth 0 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 10 monclient: _check_auth_tickets 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 1 -- 172.21.15.192:0/1049689983 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796c02cab0 con 0x7f7974142b10 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 1 -- 172.21.15.192:0/1049689983 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c02ce40 con 0x7f7974142b10 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f797a24f640 5 monclient: authenticate success, global_id 4112 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f797a24f640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f797a24f640 4 set_mon_vals no callback set 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.477+0000 7f79727fc640 10 monclient: dump: 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:05.480 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:05.481 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.481 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.481 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:05.481 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.481 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:05.481 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:05.481 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:05.481 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.484 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 10 monclient: shutdown 2023-12-15T17:01:05.484 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:05.484 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 1 -- 172.21.15.192:0/1049689983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 msgr2=0x7f797413f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:05.484 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 1 --2- 172.21.15.192:0/1049689983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f797413f080 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f79740b5270 tx=0x7f796c02bdb0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.484 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 1 -- 172.21.15.192:0/1049689983 shutdown_connections 2023-12-15T17:01:05.484 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 1 --2- 172.21.15.192:0/1049689983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f797413f080 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 1 --2- 172.21.15.192:0/1049689983 >> v2:172.21.15.192:3300/0 conn(0x7f79740b2260 0x7f79741425d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 1 --2- 172.21.15.192:0/1049689983 >> v2:172.21.15.69:3300/0 conn(0x7f79740a8a00 0x7f79740a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 1 -- 172.21.15.192:0/1049689983 >> 172.21.15.192:0/1049689983 conn(0x7f7974061a90 msgr2=0x7f79740b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 1 -- 172.21.15.192:0/1049689983 shutdown_connections 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 1 -- 172.21.15.192:0/1049689983 wait complete. 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 10 monclient: build_initial_monmap 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.478+0000 7f797a24f640 10 monclient: monmap: 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.479+0000 7f797a24f640 1 librados: starting msgr at 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.479+0000 7f797a24f640 1 librados: starting objecter 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.479+0000 7f797a24f640 1 Processor -- start 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.479+0000 7f797a24f640 1 -- start start 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.479+0000 7f797a24f640 1 librados: setting wanted keys 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.479+0000 7f797a24f640 1 librados: calling monclient init 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.479+0000 7f797a24f640 10 monclient: init 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79740a8a00 0x7f79741daeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 10 monclient(hunting): picked mon.noname-b con 0x7f79740a8a00 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:05.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79740b2260 0x7f79740b7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 10 monclient(hunting): picked mon.noname-a con 0x7f79740b2260 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f79740b5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 10 monclient(hunting): picked mon.noname-c con 0x7f7974142b10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79740b75f0 con 0x7f79740b2260 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7974145480 con 0x7f79740a8a00 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79741da990 con 0x7f7974142b10 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f797a24f640 10 monclient(hunting): authenticate will time out at 1374.763454s 2023-12-15T17:01:05.487 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f7973fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f79740b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f79737fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79740a8a00 0x7f79741daeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f7972ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79740b2260 0x7f79740b7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f7973fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f79740b5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.192:45266/0 (socket says 172.21.15.192:45266) 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f7973fff640 1 -- 172.21.15.192:0/2119018251 learned_addr learned my addr 172.21.15.192:0/2119018251 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f7973fff640 10 monclient(hunting): get_auth_request con 0x7f7974142b10 auth_method 0 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f7973fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f7973fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f7973fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f79737fe640 10 monclient(hunting): get_auth_request con 0x7f79740a8a00 auth_method 0 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f79737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f79737fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.480+0000 7f79737fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f7972ffd640 10 monclient(hunting): get_auth_request con 0x7f79740b2260 auth_method 0 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f7972ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:05.488 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f7972ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:05.489 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f7972ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:05.489 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f7973fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:05.489 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f7973fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:05.489 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f7973fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:05.489 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f79737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:05.489 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f79737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:05.489 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f79737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:05.489 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f7972ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:05.489 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f7972ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:05.489 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.481+0000 7f7972ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:05.490 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.484+0000 7f7973fff640 10 monclient(hunting): handle_auth_done global_id 4115 payload 824 2023-12-15T17:01:05.490 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.484+0000 7f7973fff640 1 -- 172.21.15.192:0/2119018251 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79740a8a00 msgr2=0x7f79741daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:05.490 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.484+0000 7f7973fff640 1 --2- 172.21.15.192:0/2119018251 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79740a8a00 0x7f79741daeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.490 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.484+0000 7f7973fff640 1 -- 172.21.15.192:0/2119018251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79740b2260 msgr2=0x7f79740b7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:05.490 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.484+0000 7f7973fff640 1 --2- 172.21.15.192:0/2119018251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79740b2260 0x7f79740b7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.490 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.484+0000 7f7973fff640 10 monclient: _finish_hunting 0 2023-12-15T17:01:05.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.484+0000 7f7973fff640 1 monclient: found mon.noname-c 2023-12-15T17:01:05.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.484+0000 7f7973fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:05.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.484+0000 7f7973fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:01:05.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.484+0000 7f7973fff640 1 -- 172.21.15.192:0/2119018251 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c009b30 con 0x7f7974142b10 2023-12-15T17:01:05.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.486+0000 7f7973fff640 1 --2- 172.21.15.192:0/2119018251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f79740b5410 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f796c036040 tx=0x7f796c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:05.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.486+0000 7f795bfff640 10 client.?.objecter ms_handle_connect 0x7f7974142b10 2023-12-15T17:01:05.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.486+0000 7f795bfff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:05.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 1 -- 172.21.15.192:0/2119018251 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c040670 con 0x7f7974142b10 2023-12-15T17:01:05.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:05.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 10 monclient: dump: 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 10 monclient: _finish_auth 0 2023-12-15T17:01:05.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 10 monclient: _check_auth_tickets 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 1 -- 172.21.15.192:0/2119018251 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796c0407d0 con 0x7f7974142b10 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 1 -- 172.21.15.192:0/2119018251 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c040b80 con 0x7f7974142b10 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f797a24f640 5 monclient: authenticate success, global_id 4115 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f797a24f640 10 monclient: _renew_subs 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f797a24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f797a24f640 1 -- 172.21.15.192:0/2119018251 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79741da620 con 0x7f7974142b10 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f7971ffb640 4 set_mon_vals no callback set 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f795bfff640 10 monclient: dump: 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.494 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f797a24f640 10 client.4115.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f797a24f640 10 monclient: _renew_subs 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f797a24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f797a24f640 1 -- 172.21.15.192:0/2119018251 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79740b7f50 con 0x7f7974142b10 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.489+0000 7f797a24f640 1 librados: init done 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.491+0000 7f797a24f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.491+0000 7f797a24f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:05.495 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.491+0000 7f797a24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:05.496 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.491+0000 7f797a24f640 1 -- 172.21.15.192:0/2119018251 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7974067ac0 con 0x7f7974142b10 2023-12-15T17:01:05.496 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.492+0000 7f795bfff640 1 -- 172.21.15.192:0/2119018251 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f796c0527b0 con 0x7f7974142b10 2023-12-15T17:01:05.496 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.492+0000 7f795bfff640 1 -- 172.21.15.192:0/2119018251 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f796c03e840 con 0x7f7974142b10 2023-12-15T17:01:05.496 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.492+0000 7f795bfff640 10 client.4115.objecter ms_dispatch 0x7f7974000b90 osd_map(4..4 src has 1..4) v4 2023-12-15T17:01:05.496 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.492+0000 7f795bfff640 3 client.4115.objecter handle_osd_map got epochs [4,4] > 0 2023-12-15T17:01:05.496 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.492+0000 7f795bfff640 3 client.4115.objecter handle_osd_map decoding full epoch 4 2023-12-15T17:01:05.497 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.492+0000 7f795bfff640 20 client.4115.objecter dump_active .. 0 homeless 2023-12-15T17:01:05.528 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:05.528+0000 7f58bb614040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-15T17:01:05.592 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.591+0000 7f795bfff640 1 -- 172.21.15.192:0/2119018251 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f796c044020 con 0x7f7974142b10 2023-12-15T17:01:05.592 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.591+0000 7f795bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:05.592 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.591+0000 7f795bfff640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:05.595 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:05.594+0000 7f70b0848040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-15T17:01:05.639 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:05.637+0000 7f70b0848040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-15T17:01:05.664 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.663+0000 7f797a24f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eaac40b5-9fc0-4e13-a322-763a9f3b685f", "id": 2}] 2023-12-15T17:01:05.664 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.663+0000 7f797a24f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eaac40b5-9fc0-4e13-a322-763a9f3b685f", "id": 2}] 2023-12-15T17:01:05.665 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.663+0000 7f797a24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:05.665 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.663+0000 7f797a24f640 1 -- 172.21.15.192:0/2119018251 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "eaac40b5-9fc0-4e13-a322-763a9f3b685f", "id": 2} v 0) v1 -- 0x7f79740b80e0 con 0x7f7974142b10 2023-12-15T17:01:05.730 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.728+0000 7f795bfff640 1 -- 172.21.15.192:0/2119018251 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eaac40b5-9fc0-4e13-a322-763a9f3b685f", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f796c03e420 con 0x7f7974142b10 2023-12-15T17:01:05.730 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.729+0000 7f795bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eaac40b5-9fc0-4e13-a322-763a9f3b685f", "id": 2}] 2023-12-15T17:01:05.730 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.729+0000 7f795bfff640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:05.731 INFO:teuthology.orchestra.run.smithi192.stdout:2 2023-12-15T17:01:05.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.729+0000 7f797a24f640 10 librados: watch_flush enter 2023-12-15T17:01:05.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.729+0000 7f797a24f640 10 librados: watch_flush exit 2023-12-15T17:01:05.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 20 client.4115.objecter shutdown clearing up homeless session... 2023-12-15T17:01:05.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 10 client.4115.objecter successfully canceled tick 2023-12-15T17:01:05.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 10 monclient: shutdown 2023-12-15T17:01:05.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:05.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 1 -- 172.21.15.192:0/2119018251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 msgr2=0x7f79740b5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:05.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 1 --2- 172.21.15.192:0/2119018251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f79740b5410 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f796c036040 tx=0x7f796c004a40 comp rx=0 tx=0).stop 2023-12-15T17:01:05.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 1 -- 172.21.15.192:0/2119018251 shutdown_connections 2023-12-15T17:01:05.733 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 1 --2- 172.21.15.192:0/2119018251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7974142b10 0x7f79740b5410 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.733 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 1 --2- 172.21.15.192:0/2119018251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79740b2260 0x7f79740b7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.733 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 1 --2- 172.21.15.192:0/2119018251 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79740a8a00 0x7f79741daeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.733 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 1 -- 172.21.15.192:0/2119018251 >> 172.21.15.192:0/2119018251 conn(0x7f7974061a90 msgr2=0x7f797413e510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:05.733 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 1 -- 172.21.15.192:0/2119018251 shutdown_connections 2023-12-15T17:01:05.733 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 1 -- 172.21.15.192:0/2119018251 wait complete. 2023-12-15T17:01:05.733 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.730+0000 7f797a24f640 1 librados: shutdown 2023-12-15T17:01:05.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad 3 2023-12-15T17:01:05.788 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:05.787+0000 7f70b0848040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.840+0000 7f26d85c7640 10 monclient: get_monmap_and_config 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.840+0000 7f26d85c7640 10 monclient: build_initial_monmap 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.840+0000 7f26d85c7640 10 monclient: monmap: 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:01:05.841615+0000 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:01:05.841615+0000 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:05.842 INFO:teuthology.orchestra.run.smithi192.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:05.843 INFO:teuthology.orchestra.run.smithi192.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:05.843 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.844 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.841+0000 7f26d85c7640 1 Processor -- start 2023-12-15T17:01:05.844 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.841+0000 7f26d85c7640 1 -- start start 2023-12-15T17:01:05.844 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.841+0000 7f26d85c7640 10 monclient: init 2023-12-15T17:01:05.844 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26d006ccb0 addr v1:172.21.15.69:6789/0 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f26d013f480 0x7f26d00ae200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 10 monclient(hunting): picked mon.noname-a con 0x7f26d013f480 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d00ae740 0x7f26d014c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 10 monclient(hunting): picked mon.noname-b con 0x7f26d00ae740 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d00adde0 con 0x7f26d006ccb0 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d006d090 con 0x7f26d013f480 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d013f8c0 con 0x7f26d00ae740 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d85c7640 10 monclient(hunting): authenticate will time out at 1375.125542s 2023-12-15T17:01:05.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.842+0000 7f26d6b64640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d00ae740 0x7f26d014c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:05.846 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.843+0000 7f26d6b64640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d00ae740 0x7f26d014c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.192:45278/0 (socket says 172.21.15.192:45278) 2023-12-15T17:01:05.846 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.843+0000 7f26d6b64640 1 -- 172.21.15.192:0/2299493102 learned_addr learned my addr 172.21.15.192:0/2299493102 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:05.846 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.843+0000 7f26d6b64640 10 monclient(hunting): get_auth_request con 0x7f26d00ae740 auth_method 0 2023-12-15T17:01:05.846 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.843+0000 7f26d6b64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:05.846 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.843+0000 7f26d6b64640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:05.846 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.843+0000 7f26d6b64640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:05.846 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.843+0000 7f26d6b64640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:05.846 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.843+0000 7f26d6b64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:05.846 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.843+0000 7f26d6b64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:05.847 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.844+0000 7f26d5b62640 1 --2- 172.21.15.192:0/2299493102 >> v2:172.21.15.69:3300/0 conn(0x7f26d013f480 0x7f26d00ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:05.847 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.844+0000 7f26d5b62640 10 monclient(hunting): get_auth_request con 0x7f26d013f480 auth_method 0 2023-12-15T17:01:05.847 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.844+0000 7f26d5b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:05.847 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.844+0000 7f26d5b62640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:05.847 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.844+0000 7f26d5b62640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:05.847 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.845+0000 7f26d6b64640 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2023-12-15T17:01:05.847 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.845+0000 7f26d6b64640 1 -- 172.21.15.192:0/2299493102 >> v2:172.21.15.69:3300/0 conn(0x7f26d013f480 msgr2=0x7f26d00ae200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:05.847 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.845+0000 7f26d6b64640 1 --2- 172.21.15.192:0/2299493102 >> v2:172.21.15.69:3300/0 conn(0x7f26d013f480 0x7f26d00ae200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.847 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.845+0000 7f26d6b64640 1 -- 172.21.15.192:0/2299493102 >> v1:172.21.15.69:6789/0 conn(0x7f26d006ccb0 legacy=0x7f26d013ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:05.847 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.845+0000 7f26d6b64640 10 monclient: _finish_hunting 0 2023-12-15T17:01:05.847 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.845+0000 7f26d6b64640 1 monclient: found mon.noname-b 2023-12-15T17:01:05.848 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.845+0000 7f26d6b64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:05.848 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.845+0000 7f26d6b64640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:01:05.848 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.845+0000 7f26d6b64640 1 -- 172.21.15.192:0/2299493102 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d00b0370 con 0x7f26d00ae740 2023-12-15T17:01:05.848 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.846+0000 7f26d6b64640 1 --2- 172.21.15.192:0/2299493102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d00ae740 0x7f26d014c690 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f26cc009e50 tx=0x7f26cc02bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d81c972f19b7ac1 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 1 -- 172.21.15.192:0/2299493102 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc02c950 con 0x7f26d00ae740 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 10 monclient: dump: 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:05.853 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 10 monclient: _finish_auth 0 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 10 monclient: _check_auth_tickets 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 1 -- 172.21.15.192:0/2299493102 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26cc02cab0 con 0x7f26d00ae740 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d85c7640 5 monclient: authenticate success, global_id 4118 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d85c7640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d85c7640 4 set_mon_vals no callback set 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 1 -- 172.21.15.192:0/2299493102 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc02ce40 con 0x7f26d00ae740 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.851+0000 7f26d5361640 10 monclient: dump: 2023-12-15T17:01:05.854 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:05.855 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:05.855 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.855 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.855 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:05.855 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.855 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:05.855 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:05.855 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:05.855 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.852+0000 7f26d85c7640 10 monclient: shutdown 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.852+0000 7f26d85c7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.852+0000 7f26d85c7640 1 -- 172.21.15.192:0/2299493102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d00ae740 msgr2=0x7f26d014c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.852+0000 7f26d85c7640 1 --2- 172.21.15.192:0/2299493102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d00ae740 0x7f26d014c690 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f26cc009e50 tx=0x7f26cc02bdb0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.852+0000 7f26d85c7640 1 -- 172.21.15.192:0/2299493102 shutdown_connections 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.852+0000 7f26d85c7640 1 --2- 172.21.15.192:0/2299493102 >> v2:172.21.15.69:3300/0 conn(0x7f26d013f480 0x7f26d00ae200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.852+0000 7f26d85c7640 1 --2- 172.21.15.192:0/2299493102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d00ae740 0x7f26d014c690 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.852+0000 7f26d85c7640 1 -- 172.21.15.192:0/2299493102 >> 172.21.15.192:0/2299493102 conn(0x7f26d00b1680 msgr2=0x7f26d00b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.852+0000 7f26d85c7640 1 -- 172.21.15.192:0/2299493102 shutdown_connections 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.852+0000 7f26d85c7640 1 -- 172.21.15.192:0/2299493102 wait complete. 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 10 monclient: build_initial_monmap 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 10 monclient: monmap: 2023-12-15T17:01:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 1 librados: starting msgr at 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 1 librados: starting objecter 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 1 Processor -- start 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 1 -- start start 2023-12-15T17:01:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 1 librados: setting wanted keys 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 1 librados: calling monclient init 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 10 monclient: init 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d006ccb0 0x7f26d01caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.853+0000 7f26d85c7640 10 monclient(hunting): picked mon.noname-c con 0x7f26d006ccb0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26d00ae740 0x7f26d01cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 10 monclient(hunting): picked mon.noname-a con 0x7f26d00ae740 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26d013f480 0x7f26d01d7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 10 monclient(hunting): picked mon.noname-b con 0x7f26d013f480 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d01d76a0 con 0x7f26d00ae740 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d01c9ef0 con 0x7f26d013f480 2023-12-15T17:01:05.860 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d01c81b0 con 0x7f26d006ccb0 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.854+0000 7f26d85c7640 10 monclient(hunting): authenticate will time out at 1375.137380s 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.855+0000 7f26d5b62640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26d00ae740 0x7f26d01cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.856+0000 7f26d6363640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d006ccb0 0x7f26d01caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.856+0000 7f26d5b62640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26d00ae740 0x7f26d01cb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.192:42580/0 (socket says 172.21.15.192:42580) 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.856+0000 7f26d5b62640 1 -- 172.21.15.192:0/1954871246 learned_addr learned my addr 172.21.15.192:0/1954871246 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.856+0000 7f26d5b62640 10 monclient(hunting): get_auth_request con 0x7f26d00ae740 auth_method 0 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.856+0000 7f26d5b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.856+0000 7f26d5b62640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.856+0000 7f26d5b62640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.856+0000 7f26d6b64640 1 --2- 172.21.15.192:0/1954871246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26d013f480 0x7f26d01d7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.856+0000 7f26d5b62640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:05.861 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.856+0000 7f26d5b62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:05.862 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.856+0000 7f26d5b62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:05.863 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.857+0000 7f26d6363640 10 monclient(hunting): get_auth_request con 0x7f26d006ccb0 auth_method 0 2023-12-15T17:01:05.863 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.857+0000 7f26d6363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:05.863 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.857+0000 7f26d6363640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:05.863 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.857+0000 7f26d6363640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:05.863 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.858+0000 7f26d6b64640 10 monclient(hunting): get_auth_request con 0x7f26d013f480 auth_method 0 2023-12-15T17:01:05.863 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.858+0000 7f26d6b64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:05.863 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.858+0000 7f26d6b64640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:05.863 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.858+0000 7f26d6b64640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:05.863 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.858+0000 7f26d6b64640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:05.863 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.858+0000 7f26d6b64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.858+0000 7f26d6b64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d6363640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d6363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d6363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d5b62640 10 monclient(hunting): handle_auth_done global_id 4131 payload 824 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d5b62640 1 -- 172.21.15.192:0/1954871246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d006ccb0 msgr2=0x7f26d01caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d5b62640 1 --2- 172.21.15.192:0/1954871246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d006ccb0 0x7f26d01caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d5b62640 1 -- 172.21.15.192:0/1954871246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26d013f480 msgr2=0x7f26d01d7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d5b62640 1 --2- 172.21.15.192:0/1954871246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26d013f480 0x7f26d01d7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d5b62640 10 monclient: _finish_hunting 0 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d5b62640 1 monclient: found mon.noname-a 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d5b62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d5b62640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:01:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.859+0000 7f26d5b62640 1 -- 172.21.15.192:0/1954871246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc009b30 con 0x7f26d00ae740 2023-12-15T17:01:05.865 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.863+0000 7f26d5b62640 1 --2- 172.21.15.192:0/1954871246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26d00ae740 0x7f26d01cb840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f26c000d5f0 tx=0x7f26c000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:05.865 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.863+0000 7f26c6ffd640 10 client.?.objecter ms_handle_connect 0x7f26d00ae740 2023-12-15T17:01:05.865 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.863+0000 7f26c6ffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:05.870 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 1 -- 172.21.15.192:0/1954871246 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c00125f0 con 0x7f26d00ae740 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 10 monclient: dump: 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.871 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 10 monclient: _finish_auth 0 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 1 -- 172.21.15.192:0/1954871246 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c0012750 con 0x7f26d00ae740 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26d85c7640 5 monclient: authenticate success, global_id 4131 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26d85c7640 10 monclient: _renew_subs 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26d85c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26d85c7640 1 -- 172.21.15.192:0/1954871246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d01cbd80 con 0x7f26d00ae740 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 1 -- 172.21.15.192:0/1954871246 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0012b00 con 0x7f26d00ae740 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26c6ffd640 10 monclient: dump: 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:05.872 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26d4b60640 4 set_mon_vals no callback set 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26d85c7640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26d85c7640 10 monclient: _renew_subs 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26d85c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26d85c7640 1 -- 172.21.15.192:0/1954871246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d01d82a0 con 0x7f26d00ae740 2023-12-15T17:01:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.869+0000 7f26d85c7640 1 librados: init done 2023-12-15T17:01:05.874 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.870+0000 7f26d85c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:05.874 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.870+0000 7f26d85c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:05.874 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.870+0000 7f26d85c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:05.874 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:05.870+0000 7f26d85c7640 1 -- 172.21.15.192:0/1954871246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d01d7810 con 0x7f26d00ae740 2023-12-15T17:01:05.948 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:05.948+0000 7f58bb614040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-15T17:01:05.995 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:05.994+0000 7f58bb614040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-15T17:01:06.037 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.035+0000 7f26c6ffd640 1 -- 172.21.15.192:0/1954871246 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f26c001e460 con 0x7f26d00ae740 2023-12-15T17:01:06.044 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.042+0000 7f26c6ffd640 1 -- 172.21.15.192:0/1954871246 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f26c0028d10 con 0x7f26d00ae740 2023-12-15T17:01:06.044 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.043+0000 7f26c6ffd640 10 client.4131.objecter ms_dispatch 0x7f26d0000b90 osd_map(5..5 src has 1..5) v4 2023-12-15T17:01:06.045 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.043+0000 7f26c6ffd640 3 client.4131.objecter handle_osd_map got epochs [5,5] > 0 2023-12-15T17:01:06.045 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.043+0000 7f26c6ffd640 3 client.4131.objecter handle_osd_map decoding full epoch 5 2023-12-15T17:01:06.045 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.043+0000 7f26c6ffd640 20 client.4131.objecter dump_active .. 0 homeless 2023-12-15T17:01:06.045 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.043+0000 7f26c6ffd640 1 -- 172.21.15.192:0/1954871246 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f26d01d7810 con 0x7f26d00ae740 2023-12-15T17:01:06.045 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.043+0000 7f26c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:06.045 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.043+0000 7f26c6ffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:06.096 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:06.095+0000 7f58bb614040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-15T17:01:06.116 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.114+0000 7f26d85c7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad", "id": 3}] 2023-12-15T17:01:06.116 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.114+0000 7f26d85c7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad", "id": 3}] 2023-12-15T17:01:06.116 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.114+0000 7f26d85c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:06.116 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.114+0000 7f26d85c7640 1 -- 172.21.15.192:0/1954871246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad", "id": 3} v 0) v1 -- 0x7f26d01d84c0 con 0x7f26d00ae740 2023-12-15T17:01:06.158 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.156+0000 7f26c6ffd640 1 -- 172.21.15.192:0/1954871246 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f26c0028880 con 0x7f26d00ae740 2023-12-15T17:01:06.158 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.156+0000 7f26c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad", "id": 3}] 2023-12-15T17:01:06.158 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.156+0000 7f26c6ffd640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:06.158 INFO:teuthology.orchestra.run.smithi192.stdout:3 2023-12-15T17:01:06.159 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 10 librados: watch_flush enter 2023-12-15T17:01:06.159 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 10 librados: watch_flush exit 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 20 client.4131.objecter shutdown clearing up homeless session... 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 10 client.4131.objecter successfully canceled tick 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 10 monclient: shutdown 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 1 -- 172.21.15.192:0/1954871246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26d00ae740 msgr2=0x7f26d01cb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 1 --2- 172.21.15.192:0/1954871246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26d00ae740 0x7f26d01cb840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f26c000d5f0 tx=0x7f26c000dab0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 1 -- 172.21.15.192:0/1954871246 shutdown_connections 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 1 --2- 172.21.15.192:0/1954871246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26d013f480 0x7f26d01d7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 1 --2- 172.21.15.192:0/1954871246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26d00ae740 0x7f26d01cb840 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 1 --2- 172.21.15.192:0/1954871246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26d006ccb0 0x7f26d01caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.157+0000 7f26d85c7640 1 -- 172.21.15.192:0/1954871246 >> 172.21.15.192:0/1954871246 conn(0x7f26d00b1680 msgr2=0x7f26d013eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.158+0000 7f26d85c7640 1 -- 172.21.15.192:0/1954871246 shutdown_connections 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.158+0000 7f26d85c7640 1 -- 172.21.15.192:0/1954871246 wait complete. 2023-12-15T17:01:06.160 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.158+0000 7f26d85c7640 1 librados: shutdown 2023-12-15T17:01:06.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 71221e7e-8713-4fc4-bef1-c1cbe40627da 4 2023-12-15T17:01:06.202 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:06.201+0000 7f58bb614040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-15T17:01:06.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.272+0000 7f6c00397640 10 monclient: get_monmap_and_config 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.272+0000 7f6c00397640 10 monclient: build_initial_monmap 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.272+0000 7f6c00397640 10 monclient: monmap: 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:01:06.273392+0000 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:01:06.273392+0000 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:06.274 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.276 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.272+0000 7f6c00397640 1 Processor -- start 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.272+0000 7f6c00397640 1 -- start start 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.272+0000 7f6c00397640 10 monclient: init 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6bf80a8a00 0x7f6bf80a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf80a8a00 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6bf80b2260 addr v1:172.21.15.69:6789/0 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6bf813f080 0x7f6bf813f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf813f080 addr v2:172.21.15.192:3300/0 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf80aa970 con 0x7f6bf80b2260 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf81421b0 con 0x7f6bf80a8a00 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:06.277 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf80aac50 con 0x7f6bf813f080 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.273+0000 7f6c00397640 10 monclient(hunting): authenticate will time out at 1375.557035s 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe934640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6bf813f080 0x7f6bf813f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe133640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6bf80a8a00 0x7f6bf80a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe934640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6bf813f080 0x7f6bf813f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52830/0 (socket says 172.21.15.192:52830) 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe934640 1 -- 172.21.15.192:0/4067865172 learned_addr learned my addr 172.21.15.192:0/4067865172 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe934640 10 monclient(hunting): get_auth_request con 0x7f6bf813f080 auth_method 0 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe934640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe934640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe934640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe133640 10 monclient(hunting): get_auth_request con 0x7f6bf80a8a00 auth_method 0 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe133640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe133640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.274+0000 7f6bfe133640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.275+0000 7f6bfe934640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:06.278 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.275+0000 7f6bfe934640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:06.279 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.275+0000 7f6bfe934640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:06.279 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.275+0000 7f6bfe133640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:06.279 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.275+0000 7f6bfe133640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:06.279 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.275+0000 7f6bfe133640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.276+0000 7f6bfe934640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2023-12-15T17:01:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.276+0000 7f6bfe934640 1 -- 172.21.15.192:0/4067865172 >> v2:172.21.15.69:3300/0 conn(0x7f6bf80a8a00 msgr2=0x7f6bf80a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.276+0000 7f6bfe934640 1 --2- 172.21.15.192:0/4067865172 >> v2:172.21.15.69:3300/0 conn(0x7f6bf80a8a00 0x7f6bf80a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.276+0000 7f6bfe934640 1 -- 172.21.15.192:0/4067865172 >> v1:172.21.15.69:6789/0 conn(0x7f6bf80b2260 legacy=0x7f6bf81425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.276+0000 7f6bfe934640 10 monclient: _finish_hunting 0 2023-12-15T17:01:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.276+0000 7f6bfe934640 1 monclient: found mon.noname-c 2023-12-15T17:01:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.276+0000 7f6bfe934640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.276+0000 7f6bfe934640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.276+0000 7f6bfe934640 1 -- 172.21.15.192:0/4067865172 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf80aaf30 con 0x7f6bf813f080 2023-12-15T17:01:06.280 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.278+0000 7f6bfe934640 1 --2- 172.21.15.192:0/4067865172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf813f080 0x7f6bf813f530 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6bf80a8670 tx=0x7f6bf402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d3d4a3190b0d12a server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:06.283 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 1 -- 172.21.15.192:0/4067865172 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf40317e0 con 0x7f6bf813f080 2023-12-15T17:01:06.283 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:06.283 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:01:06.283 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 10 monclient: dump: 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 10 monclient: _finish_auth 0 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 10 monclient: _check_auth_tickets 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 1 -- 172.21.15.192:0/4067865172 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf4031940 con 0x7f6bf813f080 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6c00397640 5 monclient: authenticate success, global_id 4138 2023-12-15T17:01:06.284 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6c00397640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6c00397640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 1 -- 172.21.15.192:0/4067865172 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4031cd0 con 0x7f6bf813f080 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6c00397640 4 set_mon_vals no callback set 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6bfd131640 10 monclient: dump: 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:06.285 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:06.286 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:06.286 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6c00397640 10 monclient: shutdown 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6c00397640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6c00397640 1 -- 172.21.15.192:0/4067865172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf813f080 msgr2=0x7f6bf813f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.282+0000 7f6c00397640 1 --2- 172.21.15.192:0/4067865172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf813f080 0x7f6bf813f530 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6bf80a8670 tx=0x7f6bf402b880 comp rx=0 tx=0).stop 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.283+0000 7f6c00397640 1 -- 172.21.15.192:0/4067865172 shutdown_connections 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.283+0000 7f6c00397640 1 --2- 172.21.15.192:0/4067865172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf813f080 0x7f6bf813f530 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.283+0000 7f6c00397640 1 --2- 172.21.15.192:0/4067865172 >> v2:172.21.15.69:3300/0 conn(0x7f6bf80a8a00 0x7f6bf80a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.283+0000 7f6c00397640 1 -- 172.21.15.192:0/4067865172 >> 172.21.15.192:0/4067865172 conn(0x7f6bf8061a90 msgr2=0x7f6bf80b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.283+0000 7f6c00397640 1 -- 172.21.15.192:0/4067865172 shutdown_connections 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.283+0000 7f6c00397640 1 -- 172.21.15.192:0/4067865172 wait complete. 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.283+0000 7f6c00397640 10 monclient: build_initial_monmap 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.283+0000 7f6c00397640 10 monclient: monmap: 2023-12-15T17:01:06.289 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 librados: starting msgr at 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 librados: starting objecter 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 Processor -- start 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 -- start start 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 librados: setting wanted keys 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 librados: calling monclient init 2023-12-15T17:01:06.290 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 10 monclient: init 2023-12-15T17:01:06.291 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:06.291 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:01:06.291 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bf80a8a00 0x7f6bf8137670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:06.291 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf80a8a00 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:06.291 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bf80b2260 0x7f6bf81d7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:06.291 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf80b2260 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:06.291 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf813f080 0x7f6bf8133a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:06.291 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf813f080 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:06.291 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:06.291 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81d7990 con 0x7f6bf80b2260 2023-12-15T17:01:06.291 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf8143c90 con 0x7f6bf813f080 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81d7be0 con 0x7f6bf80a8a00 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.284+0000 7f6c00397640 10 monclient(hunting): authenticate will time out at 1375.568165s 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe934640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf813f080 0x7f6bf8133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfd932640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bf80b2260 0x7f6bf81d7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe133640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bf80a8a00 0x7f6bf8137670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe934640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf813f080 0x7f6bf8133a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52838/0 (socket says 172.21.15.192:52838) 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe934640 1 -- 172.21.15.192:0/4010963431 learned_addr learned my addr 172.21.15.192:0/4010963431 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe934640 10 monclient(hunting): get_auth_request con 0x7f6bf813f080 auth_method 0 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe934640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe934640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe934640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfd932640 10 monclient(hunting): get_auth_request con 0x7f6bf80b2260 auth_method 0 2023-12-15T17:01:06.292 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfd932640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfd932640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfd932640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe133640 10 monclient(hunting): get_auth_request con 0x7f6bf80a8a00 auth_method 0 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe133640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe133640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.285+0000 7f6bfe133640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.286+0000 7f6bfe934640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.286+0000 7f6bfe934640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.286+0000 7f6bfe934640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.286+0000 7f6bfd932640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.286+0000 7f6bfd932640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.286+0000 7f6bfd932640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.286+0000 7f6bfe133640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:06.293 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.286+0000 7f6bfe133640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:06.294 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.286+0000 7f6bfe133640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:06.294 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.288+0000 7f6bfe934640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2023-12-15T17:01:06.294 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.288+0000 7f6bfe934640 1 -- 172.21.15.192:0/4010963431 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bf80a8a00 msgr2=0x7f6bf8137670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:06.294 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.288+0000 7f6bfe934640 1 --2- 172.21.15.192:0/4010963431 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bf80a8a00 0x7f6bf8137670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.294 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.288+0000 7f6bfe934640 1 -- 172.21.15.192:0/4010963431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bf80b2260 msgr2=0x7f6bf81d7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:06.294 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.288+0000 7f6bfe934640 1 --2- 172.21.15.192:0/4010963431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bf80b2260 0x7f6bf81d7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.295 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.288+0000 7f6bfe934640 10 monclient: _finish_hunting 0 2023-12-15T17:01:06.295 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.288+0000 7f6bfe934640 1 monclient: found mon.noname-b 2023-12-15T17:01:06.295 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.288+0000 7f6bfe934640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:06.295 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.288+0000 7f6bfe934640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.295 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.288+0000 7f6bfe934640 1 -- 172.21.15.192:0/4010963431 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf40094d0 con 0x7f6bf813f080 2023-12-15T17:01:06.295 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.290+0000 7f6bfe934640 1 --2- 172.21.15.192:0/4010963431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf813f080 0x7f6bf8133a10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40090b0 tx=0x7f6bf40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:06.295 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.290+0000 7f6beeffd640 10 client.?.objecter ms_handle_connect 0x7f6bf813f080 2023-12-15T17:01:06.295 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.290+0000 7f6beeffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:06.296 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 1 -- 172.21.15.192:0/4010963431 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4032880 con 0x7f6bf813f080 2023-12-15T17:01:06.296 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:06.296 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:06.296 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 10 monclient: dump: 2023-12-15T17:01:06.296 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:06.296 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:06.296 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.296 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.296 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:06.296 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.296 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 10 monclient: _finish_auth 0 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 10 monclient: _check_auth_tickets 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 1 -- 172.21.15.192:0/4010963431 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf40329e0 con 0x7f6bf813f080 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 1 -- 172.21.15.192:0/4010963431 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4032d90 con 0x7f6bf813f080 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6beeffd640 10 monclient: dump: 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:06.297 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6c00397640 5 monclient: authenticate success, global_id 4141 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6c00397640 10 monclient: _renew_subs 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6c00397640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6c00397640 1 -- 172.21.15.192:0/4010963431 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf81381d0 con 0x7f6bf813f080 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6bfc930640 4 set_mon_vals no callback set 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6c00397640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6c00397640 10 monclient: _renew_subs 2023-12-15T17:01:06.298 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6c00397640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.299 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6c00397640 1 -- 172.21.15.192:0/4010963431 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf8134130 con 0x7f6bf813f080 2023-12-15T17:01:06.299 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.294+0000 7f6c00397640 1 librados: init done 2023-12-15T17:01:06.299 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.295+0000 7f6c00397640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:06.299 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.295+0000 7f6c00397640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:06.300 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.295+0000 7f6c00397640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.300 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.295+0000 7f6c00397640 1 -- 172.21.15.192:0/4010963431 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf8067ac0 con 0x7f6bf813f080 2023-12-15T17:01:06.300 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.296+0000 7f6beeffd640 1 -- 172.21.15.192:0/4010963431 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f6bf404e450 con 0x7f6bf813f080 2023-12-15T17:01:06.300 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.296+0000 7f6beeffd640 1 -- 172.21.15.192:0/4010963431 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f6bf403b900 con 0x7f6bf813f080 2023-12-15T17:01:06.300 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.296+0000 7f6beeffd640 10 client.4141.objecter ms_dispatch 0x7f6bf8000b90 osd_map(6..6 src has 1..6) v4 2023-12-15T17:01:06.300 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.296+0000 7f6beeffd640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2023-12-15T17:01:06.300 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.296+0000 7f6beeffd640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2023-12-15T17:01:06.300 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.296+0000 7f6beeffd640 20 client.4141.objecter dump_active .. 0 homeless 2023-12-15T17:01:06.346 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:06.345+0000 7f58bb614040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-15T17:01:06.388 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:06.387+0000 7f58bb614040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-15T17:01:06.410 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.408+0000 7f6beeffd640 1 -- 172.21.15.192:0/4010963431 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f6bf402e070 con 0x7f6bf813f080 2023-12-15T17:01:06.410 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.408+0000 7f6beeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:06.410 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.408+0000 7f6beeffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:06.483 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.481+0000 7f6c00397640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "71221e7e-8713-4fc4-bef1-c1cbe40627da", "id": 4}] 2023-12-15T17:01:06.483 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.481+0000 7f6c00397640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "71221e7e-8713-4fc4-bef1-c1cbe40627da", "id": 4}] 2023-12-15T17:01:06.483 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.481+0000 7f6c00397640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.483 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.481+0000 7f6c00397640 1 -- 172.21.15.192:0/4010963431 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "71221e7e-8713-4fc4-bef1-c1cbe40627da", "id": 4} v 0) v1 -- 0x7f6bf8075b60 con 0x7f6bf813f080 2023-12-15T17:01:06.531 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:06.530+0000 7f58bb614040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-15T17:01:06.555 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.553+0000 7f6beeffd640 1 -- 172.21.15.192:0/4010963431 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "71221e7e-8713-4fc4-bef1-c1cbe40627da", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6bf403bb10 con 0x7f6bf813f080 2023-12-15T17:01:06.555 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.553+0000 7f6beeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "71221e7e-8713-4fc4-bef1-c1cbe40627da", "id": 4}] 2023-12-15T17:01:06.555 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.553+0000 7f6beeffd640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:06.555 INFO:teuthology.orchestra.run.smithi192.stdout:4 2023-12-15T17:01:06.557 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 10 librados: watch_flush enter 2023-12-15T17:01:06.557 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 10 librados: watch_flush exit 2023-12-15T17:01:06.557 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 20 client.4141.objecter shutdown clearing up homeless session... 2023-12-15T17:01:06.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 10 client.4141.objecter successfully canceled tick 2023-12-15T17:01:06.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 10 monclient: shutdown 2023-12-15T17:01:06.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:06.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 1 -- 172.21.15.192:0/4010963431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf813f080 msgr2=0x7f6bf8133a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:06.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 1 --2- 172.21.15.192:0/4010963431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf813f080 0x7f6bf8133a10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40090b0 tx=0x7f6bf40046c0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 1 -- 172.21.15.192:0/4010963431 shutdown_connections 2023-12-15T17:01:06.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 1 --2- 172.21.15.192:0/4010963431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf813f080 0x7f6bf8133a10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.559 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 1 --2- 172.21.15.192:0/4010963431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bf80b2260 0x7f6bf81d7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.559 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 1 --2- 172.21.15.192:0/4010963431 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bf80a8a00 0x7f6bf8137670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.559 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 1 -- 172.21.15.192:0/4010963431 >> 172.21.15.192:0/4010963431 conn(0x7f6bf8061a90 msgr2=0x7f6bf8138ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:06.559 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 1 -- 172.21.15.192:0/4010963431 shutdown_connections 2023-12-15T17:01:06.559 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 1 -- 172.21.15.192:0/4010963431 wait complete. 2023-12-15T17:01:06.559 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.554+0000 7f6c00397640 1 librados: shutdown 2023-12-15T17:01:06.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 2c9feecc-d3b5-429d-98fe-e6eddc91eb8f 5 2023-12-15T17:01:06.664 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:06.663+0000 7f70b0848040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-15T17:01:06.666 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.664+0000 7f7aaeba0640 10 monclient: get_monmap_and_config 2023-12-15T17:01:06.666 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.664+0000 7f7aaeba0640 10 monclient: build_initial_monmap 2023-12-15T17:01:06.666 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.664+0000 7f7aaeba0640 10 monclient: monmap: 2023-12-15T17:01:06.666 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:06.667 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:06.667 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:01:06.665631+0000 2023-12-15T17:01:06.667 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:01:06.665631+0000 2023-12-15T17:01:06.667 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:06.667 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.667 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:06.667 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:06.668 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:06.668 INFO:teuthology.orchestra.run.smithi192.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:06.668 INFO:teuthology.orchestra.run.smithi192.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:06.668 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.668 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 1 Processor -- start 2023-12-15T17:01:06.668 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 1 -- start start 2023-12-15T17:01:06.668 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 10 monclient: init 2023-12-15T17:01:06.672 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:06.672 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-12-15T17:01:06.672 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7aa80a87c0 0x7f7aa80a8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:06.672 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 10 monclient(hunting): picked mon.noname-c con 0x7f7aa80a87c0 addr v2:172.21.15.192:3300/0 2023-12-15T17:01:06.672 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7aa80a9a00 0x7f7aa81422d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:06.672 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 10 monclient(hunting): picked mon.noname-b con 0x7f7aa80a9a00 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7aa8142810 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa80aaa60 con 0x7f7aa8142810 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa8141eb0 con 0x7f7aa80a87c0 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80aacd0 con 0x7f7aa80a9a00 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.665+0000 7f7aaeba0640 10 monclient(hunting): authenticate will time out at 1375.949245s 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.666+0000 7f7aadb9e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7aa80a87c0 0x7f7aa80a8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.666+0000 7f7aad39d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7aa80a9a00 0x7f7aa81422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.666+0000 7f7aadb9e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7aa80a87c0 0x7f7aa80a8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52842/0 (socket says 172.21.15.192:52842) 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.666+0000 7f7aadb9e640 1 -- 172.21.15.192:0/78544004 learned_addr learned my addr 172.21.15.192:0/78544004 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.666+0000 7f7aadb9e640 10 monclient(hunting): get_auth_request con 0x7f7aa80a87c0 auth_method 0 2023-12-15T17:01:06.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.666+0000 7f7aadb9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.666+0000 7f7aadb9e640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.666+0000 7f7aadb9e640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.667+0000 7f7aad39d640 10 monclient(hunting): get_auth_request con 0x7f7aa80a9a00 auth_method 0 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.667+0000 7f7aad39d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.667+0000 7f7aad39d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.667+0000 7f7aad39d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.667+0000 7f7aadb9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.667+0000 7f7aadb9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.667+0000 7f7aadb9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.667+0000 7f7aad39d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.667+0000 7f7aad39d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.667+0000 7f7aad39d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.669+0000 7f7aadb9e640 10 monclient(hunting): handle_auth_done global_id 4144 payload 274 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.669+0000 7f7aadb9e640 1 -- 172.21.15.192:0/78544004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7aa80a9a00 msgr2=0x7f7aa81422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:06.674 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.669+0000 7f7aadb9e640 1 --2- 172.21.15.192:0/78544004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7aa80a9a00 0x7f7aa81422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.675 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.669+0000 7f7aadb9e640 1 -- 172.21.15.192:0/78544004 >> v1:172.21.15.192:6789/0 conn(0x7f7aa8142810 legacy=0x7f7aa813ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:06.675 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.669+0000 7f7aadb9e640 10 monclient: _finish_hunting 0 2023-12-15T17:01:06.675 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.669+0000 7f7aadb9e640 1 monclient: found mon.noname-c 2023-12-15T17:01:06.675 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.669+0000 7f7aadb9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:06.675 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.669+0000 7f7aadb9e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.675 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.669+0000 7f7aadb9e640 1 -- 172.21.15.192:0/78544004 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa80b1f70 con 0x7f7aa80a87c0 2023-12-15T17:01:06.675 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.674+0000 7f7aadb9e640 1 --2- 172.21.15.192:0/78544004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aa80a87c0 0x7f7aa80a8ba0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009870 tx=0x7f7aa402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ab16d9e42628ad2 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:06.681 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 1 -- 172.21.15.192:0/78544004 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4030840 con 0x7f7aa80a87c0 2023-12-15T17:01:06.681 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:06.681 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:01:06.681 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 10 monclient: dump: 2023-12-15T17:01:06.681 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:06.681 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:06.681 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.681 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.681 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 10 monclient: _finish_auth 0 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 10 monclient: _check_auth_tickets 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 1 -- 172.21.15.192:0/78544004 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa40309a0 con 0x7f7aa80a87c0 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aaeba0640 5 monclient: authenticate success, global_id 4144 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aaeba0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aaeba0640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aaeba0640 4 set_mon_vals no callback set 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 1 -- 172.21.15.192:0/78544004 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4030d30 con 0x7f7aa80a87c0 2023-12-15T17:01:06.682 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.679+0000 7f7aacb9c640 10 monclient: dump: 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:06.683 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.680+0000 7f7aaeba0640 10 monclient: shutdown 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.680+0000 7f7aaeba0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.680+0000 7f7aaeba0640 1 -- 172.21.15.192:0/78544004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aa80a87c0 msgr2=0x7f7aa80a8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.680+0000 7f7aaeba0640 1 --2- 172.21.15.192:0/78544004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aa80a87c0 0x7f7aa80a8ba0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009870 tx=0x7f7aa402b980 comp rx=0 tx=0).stop 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.680+0000 7f7aaeba0640 1 -- 172.21.15.192:0/78544004 shutdown_connections 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.680+0000 7f7aaeba0640 1 --2- 172.21.15.192:0/78544004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7aa80a9a00 0x7f7aa81422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.680+0000 7f7aaeba0640 1 --2- 172.21.15.192:0/78544004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aa80a87c0 0x7f7aa80a8ba0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.680+0000 7f7aaeba0640 1 -- 172.21.15.192:0/78544004 >> 172.21.15.192:0/78544004 conn(0x7f7aa8061890 msgr2=0x7f7aa80b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.680+0000 7f7aaeba0640 1 -- 172.21.15.192:0/78544004 shutdown_connections 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.680+0000 7f7aaeba0640 1 -- 172.21.15.192:0/78544004 wait complete. 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.681+0000 7f7aaeba0640 10 monclient: build_initial_monmap 2023-12-15T17:01:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.681+0000 7f7aaeba0640 10 monclient: monmap: 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.681+0000 7f7aaeba0640 1 librados: starting msgr at 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.681+0000 7f7aaeba0640 1 librados: starting objecter 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.681+0000 7f7aaeba0640 1 Processor -- start 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.681+0000 7f7aaeba0640 1 -- start start 2023-12-15T17:01:06.688 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.681+0000 7f7aaeba0640 1 librados: setting wanted keys 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.681+0000 7f7aaeba0640 1 librados: calling monclient init 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.681+0000 7f7aaeba0640 10 monclient: init 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aa80a87c0 0x7f7aa813bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 10 monclient(hunting): picked mon.noname-a con 0x7f7aa80a87c0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7aa80a9a00 0x7f7aa8138e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 10 monclient(hunting): picked mon.noname-c con 0x7f7aa80a9a00 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aa8142810 0x7f7aa8139bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 10 monclient(hunting): picked mon.noname-b con 0x7f7aa8142810 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa8139380 con 0x7f7aa80a87c0 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:06.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa813b210 con 0x7f7aa8142810 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa8137f20 con 0x7f7aa80a9a00 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aaeba0640 10 monclient(hunting): authenticate will time out at 1375.965795s 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.682+0000 7f7aae39f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aa8142810 0x7f7aa8139bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aadb9e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aa80a87c0 0x7f7aa813bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aad39d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7aa80a9a00 0x7f7aa8138e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aae39f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aa8142810 0x7f7aa8139bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52852/0 (socket says 172.21.15.192:52852) 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aae39f640 1 -- 172.21.15.192:0/1917391003 learned_addr learned my addr 172.21.15.192:0/1917391003 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aae39f640 10 monclient(hunting): get_auth_request con 0x7f7aa8142810 auth_method 0 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aae39f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aae39f640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aae39f640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aadb9e640 10 monclient(hunting): get_auth_request con 0x7f7aa80a87c0 auth_method 0 2023-12-15T17:01:06.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aadb9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:06.691 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aadb9e640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:06.691 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aadb9e640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:06.691 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aad39d640 10 monclient(hunting): get_auth_request con 0x7f7aa80a9a00 auth_method 0 2023-12-15T17:01:06.691 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aad39d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:06.691 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aad39d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:06.691 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aad39d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:06.691 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aae39f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:06.691 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aae39f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:06.691 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aae39f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:06.691 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aadb9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:06.692 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aadb9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:06.692 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.683+0000 7f7aadb9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:06.692 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.684+0000 7f7aad39d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:06.692 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.684+0000 7f7aad39d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:06.692 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.684+0000 7f7aad39d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:06.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.686+0000 7f7aae39f640 10 monclient(hunting): handle_auth_done global_id 4150 payload 824 2023-12-15T17:01:06.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.686+0000 7f7aae39f640 1 -- 172.21.15.192:0/1917391003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7aa80a9a00 msgr2=0x7f7aa8138e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:06.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.686+0000 7f7aae39f640 1 --2- 172.21.15.192:0/1917391003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7aa80a9a00 0x7f7aa8138e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.686+0000 7f7aae39f640 1 -- 172.21.15.192:0/1917391003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aa80a87c0 msgr2=0x7f7aa813bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:06.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.686+0000 7f7aae39f640 1 --2- 172.21.15.192:0/1917391003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aa80a87c0 0x7f7aa813bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:06.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.686+0000 7f7aae39f640 10 monclient: _finish_hunting 0 2023-12-15T17:01:06.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.686+0000 7f7aae39f640 1 monclient: found mon.noname-b 2023-12-15T17:01:06.696 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.686+0000 7f7aae39f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:06.696 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.686+0000 7f7aae39f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.696 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.686+0000 7f7aae39f640 1 -- 172.21.15.192:0/1917391003 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa40094d0 con 0x7f7aa8142810 2023-12-15T17:01:06.696 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.686+0000 7f7aadb9e640 1 --2- 172.21.15.192:0/1917391003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aa80a87c0 0x7f7aa813bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-15T17:01:06.696 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.688+0000 7f7aae39f640 1 --2- 172.21.15.192:0/1917391003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aa8142810 0x7f7aa8139bc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000c6b0 tx=0x7f7aa000cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:06.696 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.688+0000 7f7a9e7fc640 10 client.?.objecter ms_handle_connect 0x7f7aa8142810 2023-12-15T17:01:06.696 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.688+0000 7f7a9e7fc640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:06.697 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9e7fc640 1 -- 172.21.15.192:0/1917391003 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0003b70 con 0x7f7aa8142810 2023-12-15T17:01:06.697 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:06.697 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:06.697 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9e7fc640 10 monclient: dump: 2023-12-15T17:01:06.697 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:06.697 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:06.697 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.698 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.698 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:06.698 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.698 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:06.698 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:06.698 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:06.698 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.699 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9e7fc640 10 monclient: _finish_auth 0 2023-12-15T17:01:06.699 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9e7fc640 10 monclient: _check_auth_tickets 2023-12-15T17:01:06.699 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:06.699 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9e7fc640 1 -- 172.21.15.192:0/1917391003 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa0003cf0 con 0x7f7aa8142810 2023-12-15T17:01:06.699 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:06.699 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9e7fc640 1 -- 172.21.15.192:0/1917391003 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0003ea0 con 0x7f7aa8142810 2023-12-15T17:01:06.699 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7aaeba0640 5 monclient: authenticate success, global_id 4150 2023-12-15T17:01:06.700 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7aaeba0640 10 monclient: _renew_subs 2023-12-15T17:01:06.700 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7aaeba0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.700 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7aaeba0640 1 -- 172.21.15.192:0/1917391003 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa813a100 con 0x7f7aa8142810 2023-12-15T17:01:06.700 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9ffff640 4 set_mon_vals no callback set 2023-12-15T17:01:06.700 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.691+0000 7f7a9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:06.700 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.692+0000 7f7a9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:06.701 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.692+0000 7f7a9e7fc640 10 monclient: dump: 2023-12-15T17:01:06.701 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:06.701 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:06.701 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.701 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:06.701 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:06.701 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:06.702 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:06.702 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:06.702 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:06.702 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:06.702 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.692+0000 7f7aaeba0640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:06.702 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.692+0000 7f7aaeba0640 10 monclient: _renew_subs 2023-12-15T17:01:06.703 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.692+0000 7f7aaeba0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.703 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.692+0000 7f7aaeba0640 1 -- 172.21.15.192:0/1917391003 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa8136370 con 0x7f7aa8142810 2023-12-15T17:01:06.703 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.692+0000 7f7aaeba0640 1 librados: init done 2023-12-15T17:01:06.703 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.693+0000 7f7aaeba0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:06.703 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.693+0000 7f7aaeba0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:06.703 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.693+0000 7f7aaeba0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.703 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.693+0000 7f7aaeba0640 1 -- 172.21.15.192:0/1917391003 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa813b730 con 0x7f7aa8142810 2023-12-15T17:01:06.844 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.843+0000 7f7a9e7fc640 1 -- 172.21.15.192:0/1917391003 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f7aa0005690 con 0x7f7aa8142810 2023-12-15T17:01:06.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.843+0000 7f7a9e7fc640 1 -- 172.21.15.192:0/1917391003 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f7aa0025cc0 con 0x7f7aa8142810 2023-12-15T17:01:06.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.843+0000 7f7a9e7fc640 10 client.4150.objecter ms_dispatch 0x7f7aa8000b90 osd_map(7..7 src has 1..7) v4 2023-12-15T17:01:06.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.843+0000 7f7a9e7fc640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2023-12-15T17:01:06.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.843+0000 7f7a9e7fc640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2023-12-15T17:01:06.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.843+0000 7f7a9e7fc640 20 client.4150.objecter dump_active .. 0 homeless 2023-12-15T17:01:06.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.843+0000 7f7a9e7fc640 1 -- 172.21.15.192:0/1917391003 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f7aa813b730 con 0x7f7aa8142810 2023-12-15T17:01:06.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.843+0000 7f7a9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:06.845 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.843+0000 7f7a9e7fc640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:06.915 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.913+0000 7f7aaeba0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2c9feecc-d3b5-429d-98fe-e6eddc91eb8f", "id": 5}] 2023-12-15T17:01:06.915 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.914+0000 7f7aaeba0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2c9feecc-d3b5-429d-98fe-e6eddc91eb8f", "id": 5}] 2023-12-15T17:01:06.915 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.914+0000 7f7aaeba0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:06.915 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:06.914+0000 7f7aaeba0640 1 -- 172.21.15.192:0/1917391003 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2c9feecc-d3b5-429d-98fe-e6eddc91eb8f", "id": 5} v 0) v1 -- 0x7f7aa81394f0 con 0x7f7aa8142810 2023-12-15T17:01:07.005 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.003+0000 7f7a9e7fc640 1 -- 172.21.15.192:0/1917391003 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2c9feecc-d3b5-429d-98fe-e6eddc91eb8f", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7aa0025830 con 0x7f7aa8142810 2023-12-15T17:01:07.005 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.003+0000 7f7a9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2c9feecc-d3b5-429d-98fe-e6eddc91eb8f", "id": 5}] 2023-12-15T17:01:07.005 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.003+0000 7f7a9e7fc640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:07.006 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:07.003+0000 7f70b0848040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-15T17:01:07.006 INFO:teuthology.orchestra.run.smithi192.stdout:5 2023-12-15T17:01:07.008 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 10 librados: watch_flush enter 2023-12-15T17:01:07.008 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 10 librados: watch_flush exit 2023-12-15T17:01:07.008 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 20 client.4150.objecter shutdown clearing up homeless session... 2023-12-15T17:01:07.009 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 10 client.4150.objecter successfully canceled tick 2023-12-15T17:01:07.009 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 10 monclient: shutdown 2023-12-15T17:01:07.009 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:07.009 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 1 -- 172.21.15.192:0/1917391003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aa8142810 msgr2=0x7f7aa8139bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:07.009 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 1 --2- 172.21.15.192:0/1917391003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aa8142810 0x7f7aa8139bc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000c6b0 tx=0x7f7aa000cb70 comp rx=0 tx=0).stop 2023-12-15T17:01:07.009 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 1 -- 172.21.15.192:0/1917391003 shutdown_connections 2023-12-15T17:01:07.010 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 1 --2- 172.21.15.192:0/1917391003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aa8142810 0x7f7aa8139bc0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.010 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 1 --2- 172.21.15.192:0/1917391003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7aa80a9a00 0x7f7aa8138e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.010 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 1 --2- 172.21.15.192:0/1917391003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aa80a87c0 0x7f7aa813bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.010 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 1 -- 172.21.15.192:0/1917391003 >> 172.21.15.192:0/1917391003 conn(0x7f7aa8061890 msgr2=0x7f7aa813ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:07.010 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 1 -- 172.21.15.192:0/1917391003 shutdown_connections 2023-12-15T17:01:07.010 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 1 -- 172.21.15.192:0/1917391003 wait complete. 2023-12-15T17:01:07.011 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.004+0000 7f7aaeba0640 1 librados: shutdown 2023-12-15T17:01:07.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7 6 2023-12-15T17:01:07.061 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:07.059+0000 7f70b0848040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-15T17:01:07.116 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.114+0000 7f55660e7640 10 monclient: get_monmap_and_config 2023-12-15T17:01:07.116 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.114+0000 7f55660e7640 10 monclient: build_initial_monmap 2023-12-15T17:01:07.116 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.114+0000 7f55660e7640 10 monclient: monmap: 2023-12-15T17:01:07.116 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:07.116 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:07.116 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:01:07.115778+0000 2023-12-15T17:01:07.116 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:01:07.115778+0000 2023-12-15T17:01:07.116 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:07.117 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.117 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:07.117 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:07.117 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:07.117 INFO:teuthology.orchestra.run.smithi192.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:07.117 INFO:teuthology.orchestra.run.smithi192.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:07.117 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.119 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.115+0000 7f55660e7640 1 Processor -- start 2023-12-15T17:01:07.119 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.115+0000 7f55660e7640 1 -- start start 2023-12-15T17:01:07.119 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.115+0000 7f55660e7640 10 monclient: init 2023-12-15T17:01:07.119 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:07.119 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-12-15T17:01:07.120 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f55600a8a00 0x7f55600a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:07.120 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 10 monclient(hunting): picked mon.noname-a con 0x7f55600a8a00 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:07.120 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f55600b2260 0x7f55601425d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:07.120 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 10 monclient(hunting): picked mon.noname-c con 0x7f55600b2260 addr v2:172.21.15.192:3300/0 2023-12-15T17:01:07.120 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5560142b10 0x7f556013f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:07.120 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 10 monclient(hunting): picked mon.noname-b con 0x7f5560142b10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:07.120 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:07.120 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55601421b0 con 0x7f55600a8a00 2023-12-15T17:01:07.120 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:07.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55600aa970 con 0x7f55600b2260 2023-12-15T17:01:07.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:07.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55600aac50 con 0x7f5560142b10 2023-12-15T17:01:07.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:07.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f55660e7640 10 monclient(hunting): authenticate will time out at 1376.399544s 2023-12-15T17:01:07.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f5556ffd640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f55600b2260 0x7f55601425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:07.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f555f7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f55600a8a00 0x7f55600a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:07.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.116+0000 7f555ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5560142b10 0x7f556013f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f5556ffd640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f55600b2260 0x7f55601425d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52854/0 (socket says 172.21.15.192:52854) 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f5556ffd640 1 -- 172.21.15.192:0/2198473907 learned_addr learned my addr 172.21.15.192:0/2198473907 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f5556ffd640 10 monclient(hunting): get_auth_request con 0x7f55600b2260 auth_method 0 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f5556ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f5556ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f5556ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f555f7fe640 10 monclient(hunting): get_auth_request con 0x7f55600a8a00 auth_method 0 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f555f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f555f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f555f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f555ffff640 10 monclient(hunting): get_auth_request con 0x7f5560142b10 auth_method 0 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f555ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f555ffff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f555ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:07.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f5556ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:07.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f5556ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:07.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f5556ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:07.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f555f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:07.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f555f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:07.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.117+0000 7f555f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:07.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.118+0000 7f555ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:07.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.118+0000 7f555ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:07.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.118+0000 7f555ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:07.124 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.119+0000 7f555f7fe640 10 monclient(hunting): handle_auth_done global_id 4143 payload 274 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.119+0000 7f555f7fe640 1 -- 172.21.15.192:0/2198473907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5560142b10 msgr2=0x7f556013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.119+0000 7f555f7fe640 1 --2- 172.21.15.192:0/2198473907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5560142b10 0x7f556013f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.119+0000 7f555f7fe640 1 -- 172.21.15.192:0/2198473907 >> v2:172.21.15.192:3300/0 conn(0x7f55600b2260 msgr2=0x7f55601425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.119+0000 7f555f7fe640 1 --2- 172.21.15.192:0/2198473907 >> v2:172.21.15.192:3300/0 conn(0x7f55600b2260 0x7f55601425d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.119+0000 7f555f7fe640 10 monclient: _finish_hunting 0 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.119+0000 7f555f7fe640 1 monclient: found mon.noname-a 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.119+0000 7f555f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.119+0000 7f555f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.119+0000 7f555f7fe640 1 -- 172.21.15.192:0/2198473907 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55600aaf30 con 0x7f55600a8a00 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.121+0000 7f555f7fe640 1 --2- 172.21.15.192:0/2198473907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55600a8a00 0x7f55600a8de0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f55600b5270 tx=0x7f555002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e9ca005ff2abb61 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.123+0000 7f555effd640 1 -- 172.21.15.192:0/2198473907 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5550030710 con 0x7f55600a8a00 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.123+0000 7f555effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.123+0000 7f555effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:01:07.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.123+0000 7f555effd640 10 monclient: dump: 2023-12-15T17:01:07.126 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:07.126 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:07.126 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.126 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.126 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:07.126 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.126 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:07.126 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:07.127 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:07.127 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.127 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.123+0000 7f555effd640 10 monclient: _finish_auth 0 2023-12-15T17:01:07.127 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.123+0000 7f555effd640 10 monclient: _check_auth_tickets 2023-12-15T17:01:07.127 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.123+0000 7f555effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:07.127 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.123+0000 7f55660e7640 5 monclient: authenticate success, global_id 4143 2023-12-15T17:01:07.127 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.123+0000 7f55660e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:07.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.124+0000 7f555effd640 1 -- 172.21.15.192:0/2198473907 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555002e050 con 0x7f55600a8a00 2023-12-15T17:01:07.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.124+0000 7f555effd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:07.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.124+0000 7f555effd640 1 -- 172.21.15.192:0/2198473907 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5550030a90 con 0x7f55600a8a00 2023-12-15T17:01:07.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.124+0000 7f55660e7640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:07.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.124+0000 7f55660e7640 4 set_mon_vals no callback set 2023-12-15T17:01:07.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.124+0000 7f555effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:07.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.124+0000 7f555effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:01:07.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.124+0000 7f555effd640 10 monclient: dump: 2023-12-15T17:01:07.134 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:07.134 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:07.134 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.134 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.134 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:07.134 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.125+0000 7f55660e7640 10 monclient: shutdown 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.125+0000 7f55660e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.125+0000 7f55660e7640 1 -- 172.21.15.192:0/2198473907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55600a8a00 msgr2=0x7f55600a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.125+0000 7f55660e7640 1 --2- 172.21.15.192:0/2198473907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55600a8a00 0x7f55600a8de0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f55600b5270 tx=0x7f555002b710 comp rx=0 tx=0).stop 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.125+0000 7f55660e7640 1 -- 172.21.15.192:0/2198473907 shutdown_connections 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.125+0000 7f55660e7640 1 --2- 172.21.15.192:0/2198473907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5560142b10 0x7f556013f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.125+0000 7f55660e7640 1 --2- 172.21.15.192:0/2198473907 >> v2:172.21.15.192:3300/0 conn(0x7f55600b2260 0x7f55601425d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.125+0000 7f55660e7640 1 --2- 172.21.15.192:0/2198473907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55600a8a00 0x7f55600a8de0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.125+0000 7f55660e7640 1 -- 172.21.15.192:0/2198473907 >> 172.21.15.192:0/2198473907 conn(0x7f5560061a90 msgr2=0x7f55600b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:07.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.125+0000 7f55660e7640 1 -- 172.21.15.192:0/2198473907 shutdown_connections 2023-12-15T17:01:07.136 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.125+0000 7f55660e7640 1 -- 172.21.15.192:0/2198473907 wait complete. 2023-12-15T17:01:07.136 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.126+0000 7f55660e7640 10 monclient: build_initial_monmap 2023-12-15T17:01:07.136 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.126+0000 7f55660e7640 10 monclient: monmap: 2023-12-15T17:01:07.136 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:07.136 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:07.136 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2023-12-15T17:01:07.137 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2023-12-15T17:01:07.137 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:07.137 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.137 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:07.137 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:07.137 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:07.137 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.138 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.126+0000 7f55660e7640 1 librados: starting msgr at 2023-12-15T17:01:07.138 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.126+0000 7f55660e7640 1 librados: starting objecter 2023-12-15T17:01:07.138 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.126+0000 7f55660e7640 1 Processor -- start 2023-12-15T17:01:07.138 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.126+0000 7f55660e7640 1 -- start start 2023-12-15T17:01:07.138 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.126+0000 7f55660e7640 1 librados: setting wanted keys 2023-12-15T17:01:07.138 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.126+0000 7f55660e7640 1 librados: calling monclient init 2023-12-15T17:01:07.139 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.126+0000 7f55660e7640 10 monclient: init 2023-12-15T17:01:07.139 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:07.139 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:01:07.139 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55600a8a00 0x7f55601da6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:07.139 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 10 monclient(hunting): picked mon.noname-b con 0x7f55600a8a00 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:07.139 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55600b2260 0x7f55600b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:07.140 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 10 monclient(hunting): picked mon.noname-a con 0x7f55600b2260 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:07.140 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5560142b10 0x7f55600b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:07.140 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 10 monclient(hunting): picked mon.noname-c con 0x7f5560142b10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:07.140 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:07.140 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600b1130 con 0x7f55600b2260 2023-12-15T17:01:07.140 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:07.141 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5560139e80 con 0x7f55600a8a00 2023-12-15T17:01:07.141 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:07.141 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556013bef0 con 0x7f5560142b10 2023-12-15T17:01:07.141 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:07.141 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.127+0000 7f55660e7640 10 monclient(hunting): authenticate will time out at 1376.410890s 2023-12-15T17:01:07.141 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555f7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55600a8a00 0x7f55601da6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:07.141 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f5556ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55600b2260 0x7f55600b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:07.142 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5560142b10 0x7f55600b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:07.142 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555f7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55600a8a00 0x7f55601da6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52856/0 (socket says 172.21.15.192:52856) 2023-12-15T17:01:07.142 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555f7fe640 1 -- 172.21.15.192:0/1078098386 learned_addr learned my addr 172.21.15.192:0/1078098386 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:07.142 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555f7fe640 10 monclient(hunting): get_auth_request con 0x7f55600a8a00 auth_method 0 2023-12-15T17:01:07.142 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:07.142 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:07.143 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:07.143 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f5556ffd640 10 monclient(hunting): get_auth_request con 0x7f55600b2260 auth_method 0 2023-12-15T17:01:07.143 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f5556ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:07.143 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f5556ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:07.143 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f5556ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:07.143 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555ffff640 10 monclient(hunting): get_auth_request con 0x7f5560142b10 auth_method 0 2023-12-15T17:01:07.143 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:07.144 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555ffff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:07.144 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:07.144 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:07.144 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:07.144 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f555f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:07.144 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f5556ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:07.144 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f5556ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:07.145 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.128+0000 7f5556ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:07.145 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.129+0000 7f555ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:07.145 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.129+0000 7f555ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:07.145 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.129+0000 7f555ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:07.145 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.131+0000 7f5556ffd640 10 monclient(hunting): handle_auth_done global_id 4146 payload 824 2023-12-15T17:01:07.145 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.131+0000 7f5556ffd640 1 -- 172.21.15.192:0/1078098386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5560142b10 msgr2=0x7f55600b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:07.146 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.131+0000 7f5556ffd640 1 --2- 172.21.15.192:0/1078098386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5560142b10 0x7f55600b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.146 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.131+0000 7f5556ffd640 1 -- 172.21.15.192:0/1078098386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55600a8a00 msgr2=0x7f55601da6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:07.146 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.131+0000 7f5556ffd640 1 --2- 172.21.15.192:0/1078098386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55600a8a00 0x7f55601da6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.146 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.131+0000 7f5556ffd640 10 monclient: _finish_hunting 0 2023-12-15T17:01:07.146 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.131+0000 7f5556ffd640 1 monclient: found mon.noname-a 2023-12-15T17:01:07.146 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.131+0000 7f5556ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:07.147 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.131+0000 7f5556ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:01:07.147 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.131+0000 7f5556ffd640 1 -- 172.21.15.192:0/1078098386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5550009400 con 0x7f55600b2260 2023-12-15T17:01:07.147 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.131+0000 7f555f7fe640 1 --2- 172.21.15.192:0/1078098386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55600a8a00 0x7f55601da6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-15T17:01:07.149 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.133+0000 7f5556ffd640 1 --2- 172.21.15.192:0/1078098386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55600b2260 0x7f55600b06b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f55600b2680 tx=0x7f554800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:07.149 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.133+0000 7f555cff9640 10 client.?.objecter ms_handle_connect 0x7f55600b2260 2023-12-15T17:01:07.150 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.133+0000 7f555cff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:07.150 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f555cff9640 1 -- 172.21.15.192:0/1078098386 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554800eeb0 con 0x7f55600b2260 2023-12-15T17:01:07.150 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f555cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:07.150 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f555cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:01:07.150 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f555cff9640 10 monclient: dump: 2023-12-15T17:01:07.150 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:07.151 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:07.151 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.151 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.151 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:07.151 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.151 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:07.151 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:07.152 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:07.152 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f555cff9640 10 monclient: _finish_auth 0 2023-12-15T17:01:07.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f555cff9640 10 monclient: _check_auth_tickets 2023-12-15T17:01:07.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f555cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:07.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f55660e7640 5 monclient: authenticate success, global_id 4146 2023-12-15T17:01:07.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f55660e7640 10 monclient: _renew_subs 2023-12-15T17:01:07.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f55660e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:07.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f55660e7640 1 -- 172.21.15.192:0/1078098386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556013c1d0 con 0x7f55600b2260 2023-12-15T17:01:07.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f55660e7640 10 client.4146.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:07.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f55660e7640 10 monclient: _renew_subs 2023-12-15T17:01:07.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f55660e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:07.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f55660e7640 1 -- 172.21.15.192:0/1078098386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55601db0e0 con 0x7f55600b2260 2023-12-15T17:01:07.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.135+0000 7f55660e7640 1 librados: init done 2023-12-15T17:01:07.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.136+0000 7f55660e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:07.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.136+0000 7f55660e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:07.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.136+0000 7f55660e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:07.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.136+0000 7f55660e7640 1 -- 172.21.15.192:0/1078098386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5560067ac0 con 0x7f55600b2260 2023-12-15T17:01:07.167 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:07.166+0000 7f70b0848040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-15T17:01:07.233 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.231+0000 7f555cff9640 1 -- 172.21.15.192:0/1078098386 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5548005990 con 0x7f55600b2260 2023-12-15T17:01:07.233 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.231+0000 7f555cff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.231+0000 7f555cff9640 1 -- 172.21.15.192:0/1078098386 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5548005d20 con 0x7f55600b2260 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.231+0000 7f555cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.231+0000 7f555cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.231+0000 7f555cff9640 10 monclient: dump: 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:07.234 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:07.235 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:07.235 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.235 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.231+0000 7f555e7fc640 4 set_mon_vals no callback set 2023-12-15T17:01:07.260 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.259+0000 7f555cff9640 1 -- 172.21.15.192:0/1078098386 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f554800c940 con 0x7f55600b2260 2023-12-15T17:01:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.259+0000 7f555cff9640 1 -- 172.21.15.192:0/1078098386 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f55480068f0 con 0x7f55600b2260 2023-12-15T17:01:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.259+0000 7f555cff9640 10 client.4146.objecter ms_dispatch 0x7f5560000b90 osd_map(8..8 src has 1..8) v4 2023-12-15T17:01:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.259+0000 7f555cff9640 3 client.4146.objecter handle_osd_map got epochs [8,8] > 0 2023-12-15T17:01:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.259+0000 7f555cff9640 3 client.4146.objecter handle_osd_map decoding full epoch 8 2023-12-15T17:01:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.259+0000 7f555cff9640 20 client.4146.objecter dump_active .. 0 homeless 2023-12-15T17:01:07.302 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.301+0000 7f555cff9640 1 -- 172.21.15.192:0/1078098386 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f5548021060 con 0x7f55600b2260 2023-12-15T17:01:07.303 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.301+0000 7f555cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:07.303 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.301+0000 7f555cff9640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:07.344 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:07.343+0000 7f58bb614040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-15T17:01:07.372 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.371+0000 7f55660e7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7", "id": 6}] 2023-12-15T17:01:07.372 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.371+0000 7f55660e7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7", "id": 6}] 2023-12-15T17:01:07.372 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.371+0000 7f55660e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:07.372 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.371+0000 7f55660e7640 1 -- 172.21.15.192:0/1078098386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7", "id": 6} v 0) v1 -- 0x7f5560075b60 con 0x7f55600b2260 2023-12-15T17:01:07.440 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.438+0000 7f555cff9640 1 -- 172.21.15.192:0/1078098386 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 2) v1 ==== 44706+0+0 (secure 0 0 0) 0x7f554800ccb0 con 0x7f55600b2260 2023-12-15T17:01:07.641 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:07.641+0000 7f58bb614040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-15T17:01:07.711 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:07.710+0000 7f58bb614040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-15T17:01:07.751 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.750+0000 7f555cff9640 1 -- 172.21.15.192:0/1078098386 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5548017020 con 0x7f55600b2260 2023-12-15T17:01:07.752 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.750+0000 7f555cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7", "id": 6}] 2023-12-15T17:01:07.752 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.750+0000 7f555cff9640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:07.752 INFO:teuthology.orchestra.run.smithi192.stdout:6 2023-12-15T17:01:07.754 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 10 librados: watch_flush enter 2023-12-15T17:01:07.754 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 10 librados: watch_flush exit 2023-12-15T17:01:07.754 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 20 client.4146.objecter shutdown clearing up homeless session... 2023-12-15T17:01:07.755 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 10 client.4146.objecter successfully canceled tick 2023-12-15T17:01:07.755 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 10 monclient: shutdown 2023-12-15T17:01:07.755 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:07.755 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 1 -- 172.21.15.192:0/1078098386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55600b2260 msgr2=0x7f55600b06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:07.755 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 1 --2- 172.21.15.192:0/1078098386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55600b2260 0x7f55600b06b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f55600b2680 tx=0x7f554800ec60 comp rx=0 tx=0).stop 2023-12-15T17:01:07.755 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 1 -- 172.21.15.192:0/1078098386 shutdown_connections 2023-12-15T17:01:07.755 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 1 --2- 172.21.15.192:0/1078098386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5560142b10 0x7f55600b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.755 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 1 --2- 172.21.15.192:0/1078098386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55600b2260 0x7f55600b06b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.755 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 1 --2- 172.21.15.192:0/1078098386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55600a8a00 0x7f55601da6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.756 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 1 -- 172.21.15.192:0/1078098386 >> 172.21.15.192:0/1078098386 conn(0x7f5560061a90 msgr2=0x7f5560073c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:07.756 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 1 -- 172.21.15.192:0/1078098386 shutdown_connections 2023-12-15T17:01:07.756 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 1 -- 172.21.15.192:0/1078098386 wait complete. 2023-12-15T17:01:07.756 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.751+0000 7f55660e7640 1 librados: shutdown 2023-12-15T17:01:07.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new ac6e8a35-6df7-48d4-b59a-1f87eceaacfa 7 2023-12-15T17:01:07.823 INFO:tasks.ceph.mgr.y.smithi069.stderr:2023-12-15T17:01:07.821+0000 7f58bb614040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-15T17:01:07.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.901+0000 7fde0f533640 10 monclient: get_monmap_and_config 2023-12-15T17:01:07.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.901+0000 7fde0f533640 10 monclient: build_initial_monmap 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.902+0000 7fde0f533640 10 monclient: monmap: 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:01:07.903081+0000 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:01:07.903081+0000 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:07.904 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.905 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.902+0000 7fde0f533640 1 Processor -- start 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.902+0000 7fde0f533640 1 -- start start 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.902+0000 7fde0f533640 10 monclient: init 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde080a8950 0x7fde0813f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 10 monclient(hunting): picked mon.noname-b con 0x7fde080a8950 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fde080a8d30 0x7fde0813f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 10 monclient(hunting): picked mon.noname-a con 0x7fde080a8d30 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde081420e0 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde080a9de0 con 0x7fde081420e0 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde08142c40 con 0x7fde080a8d30 2023-12-15T17:01:07.907 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde080aab80 con 0x7fde080a8950 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.903+0000 7fde0f533640 10 monclient(hunting): authenticate will time out at 1377.186772s 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.904+0000 7fde0dad0640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fde081420e0 0x7fde08142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:41560/0 (socket says 172.21.15.192:41560) 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.904+0000 7fde0dad0640 1 -- 172.21.15.192:0/483476948 learned_addr learned my addr 172.21.15.192:0/483476948 (peer_addr_for_me v1:172.21.15.192:0/0) 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.904+0000 7fde0d2cf640 1 --2- 172.21.15.192:0/483476948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde080a8950 0x7fde0813f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.904+0000 7fde0d2cf640 10 monclient(hunting): get_auth_request con 0x7fde080a8950 auth_method 0 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.904+0000 7fde0d2cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.904+0000 7fde0d2cf640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.904+0000 7fde0d2cf640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.904+0000 7fde0d2cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.904+0000 7fde0d2cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:07.908 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.904+0000 7fde0d2cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:07.909 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.906+0000 7fde0d2cf640 10 monclient(hunting): handle_auth_done global_id 4157 payload 274 2023-12-15T17:01:07.909 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.906+0000 7fde0d2cf640 1 -- 172.21.15.192:0/483476948 >> v2:172.21.15.69:3300/0 conn(0x7fde080a8d30 msgr2=0x7fde0813f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:07.909 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.906+0000 7fde0d2cf640 1 --2- 172.21.15.192:0/483476948 >> v2:172.21.15.69:3300/0 conn(0x7fde080a8d30 0x7fde0813f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.909 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.906+0000 7fde0d2cf640 1 -- 172.21.15.192:0/483476948 >> v1:172.21.15.192:6789/0 conn(0x7fde081420e0 legacy=0x7fde08142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:07.909 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.906+0000 7fde0d2cf640 10 monclient: _finish_hunting 0 2023-12-15T17:01:07.910 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.906+0000 7fde0d2cf640 1 monclient: found mon.noname-b 2023-12-15T17:01:07.910 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.906+0000 7fde0d2cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:07.910 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.906+0000 7fde0d2cf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:01:07.910 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.906+0000 7fde0d2cf640 1 -- 172.21.15.192:0/483476948 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde080aae60 con 0x7fde080a8950 2023-12-15T17:01:07.910 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.908+0000 7fde0d2cf640 1 --2- 172.21.15.192:0/483476948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde080a8950 0x7fde0813f040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097a0 tx=0x7fddf802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=965489d336f97762 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:07.914 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 1 -- 172.21.15.192:0/483476948 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf802f050 con 0x7fde080a8950 2023-12-15T17:01:07.914 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:07.914 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:07.914 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 10 monclient: dump: 2023-12-15T17:01:07.914 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 10 monclient: _finish_auth 0 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 10 monclient: _check_auth_tickets 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 1 -- 172.21.15.192:0/483476948 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf8031700 con 0x7fde080a8950 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:07.915 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 1 -- 172.21.15.192:0/483476948 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8031a90 con 0x7fde080a8950 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 5 monclient: authenticate success, global_id 4157 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 4 set_mon_vals no callback set 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fddfffff640 10 monclient: dump: 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:07.916 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.917 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:07.917 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:07.917 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:07.917 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 10 monclient: shutdown 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 1 -- 172.21.15.192:0/483476948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde080a8950 msgr2=0x7fde0813f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 1 --2- 172.21.15.192:0/483476948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde080a8950 0x7fde0813f040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097a0 tx=0x7fddf802b7a0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 1 -- 172.21.15.192:0/483476948 shutdown_connections 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 1 --2- 172.21.15.192:0/483476948 >> v2:172.21.15.69:3300/0 conn(0x7fde080a8d30 0x7fde0813f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 1 --2- 172.21.15.192:0/483476948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde080a8950 0x7fde0813f040 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 1 -- 172.21.15.192:0/483476948 >> 172.21.15.192:0/483476948 conn(0x7fde080b77d0 msgr2=0x7fde080b7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 1 -- 172.21.15.192:0/483476948 shutdown_connections 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.913+0000 7fde0f533640 1 -- 172.21.15.192:0/483476948 wait complete. 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.914+0000 7fde0f533640 10 monclient: build_initial_monmap 2023-12-15T17:01:07.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.914+0000 7fde0f533640 10 monclient: monmap: 2023-12-15T17:01:07.921 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2023-12-15T17:01:07.921 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:07.921 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2023-12-15T17:01:07.921 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2023-12-15T17:01:07.921 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:07.921 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.921 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:07.921 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:07.921 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:07.921 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.914+0000 7fde0f533640 1 librados: starting msgr at 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.914+0000 7fde0f533640 1 librados: starting objecter 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.914+0000 7fde0f533640 1 Processor -- start 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.914+0000 7fde0f533640 1 -- start start 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.914+0000 7fde0f533640 1 librados: setting wanted keys 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.914+0000 7fde0f533640 1 librados: calling monclient init 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.914+0000 7fde0f533640 10 monclient: init 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde080a8950 0x7fde0813a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 10 monclient(hunting): picked mon.noname-b con 0x7fde080a8950 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde080a8d30 0x7fde08131820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 10 monclient(hunting): picked mon.noname-a con 0x7fde080a8d30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde081420e0 0x7fde08131d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 10 monclient(hunting): picked mon.noname-c con 0x7fde081420e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0813c760 con 0x7fde080a8d30 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde081322a0 con 0x7fde080a8950 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0813c3c0 con 0x7fde081420e0 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0f533640 10 monclient(hunting): authenticate will time out at 1377.198643s 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0d2cf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde080a8950 0x7fde0813a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.915+0000 7fde0dad0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde081420e0 0x7fde08131d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0d2cf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde080a8950 0x7fde0813a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52916/0 (socket says 172.21.15.192:52916) 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0d2cf640 1 -- 172.21.15.192:0/2314960284 learned_addr learned my addr 172.21.15.192:0/2314960284 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0d2cf640 10 monclient(hunting): get_auth_request con 0x7fde080a8950 auth_method 0 2023-12-15T17:01:07.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0d2cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0d2cf640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0d2cf640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0dad0640 10 monclient(hunting): get_auth_request con 0x7fde081420e0 auth_method 0 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0dad0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0dad0640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0dad0640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0d2cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0d2cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0d2cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0dad0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0dad0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.916+0000 7fde0dad0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:07.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.919+0000 7fde0dad0640 10 monclient(hunting): handle_auth_done global_id 4160 payload 824 2023-12-15T17:01:07.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.919+0000 7fde0dad0640 1 -- 172.21.15.192:0/2314960284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde080a8950 msgr2=0x7fde0813a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:07.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.919+0000 7fde0dad0640 1 --2- 172.21.15.192:0/2314960284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde080a8950 0x7fde0813a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.919+0000 7fde0dad0640 1 -- 172.21.15.192:0/2314960284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde080a8d30 msgr2=0x7fde08131820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.919+0000 7fde0dad0640 1 --2- 172.21.15.192:0/2314960284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde080a8d30 0x7fde08131820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.919+0000 7fde0dad0640 10 monclient: _finish_hunting 0 2023-12-15T17:01:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.919+0000 7fde0dad0640 1 monclient: found mon.noname-c 2023-12-15T17:01:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.919+0000 7fde0dad0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.919+0000 7fde0dad0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:01:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.919+0000 7fde0dad0640 1 -- 172.21.15.192:0/2314960284 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf8009400 con 0x7fde081420e0 2023-12-15T17:01:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.919+0000 7fde0d2cf640 1 --2- 172.21.15.192:0/2314960284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde080a8950 0x7fde0813a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-15T17:01:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.921+0000 7fde0dad0640 1 --2- 172.21.15.192:0/2314960284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde081420e0 0x7fde08131d60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde0000caa0 tx=0x7fde0000cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.921+0000 7fddfdffb640 10 client.?.objecter ms_handle_connect 0x7fde081420e0 2023-12-15T17:01:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.921+0000 7fddfdffb640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:07.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 1 -- 172.21.15.192:0/2314960284 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0000ab40 con 0x7fde081420e0 2023-12-15T17:01:07.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:07.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:01:07.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 10 monclient: dump: 2023-12-15T17:01:07.930 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:07.930 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:07.930 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.930 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 10 monclient: _finish_auth 0 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 10 monclient: _check_auth_tickets 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 1 -- 172.21.15.192:0/2314960284 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0000aca0 con 0x7fde081420e0 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 1 -- 172.21.15.192:0/2314960284 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde00012600 con 0x7fde081420e0 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fde0f533640 5 monclient: authenticate success, global_id 4160 2023-12-15T17:01:07.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddfdffb640 10 monclient: dump: 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fddff7fe640 4 set_mon_vals no callback set 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fde0f533640 10 monclient: _renew_subs 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fde0f533640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:07.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fde0f533640 1 -- 172.21.15.192:0/2314960284 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde081381f0 con 0x7fde081420e0 2023-12-15T17:01:07.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fde0f533640 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:07.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fde0f533640 10 monclient: _renew_subs 2023-12-15T17:01:07.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fde0f533640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:07.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fde0f533640 1 -- 172.21.15.192:0/2314960284 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde08139940 con 0x7fde081420e0 2023-12-15T17:01:07.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.928+0000 7fde0f533640 1 librados: init done 2023-12-15T17:01:07.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.929+0000 7fde0f533640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:07.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.929+0000 7fde0f533640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:07.933 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.929+0000 7fde0f533640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:07.934 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.929+0000 7fde0f533640 1 -- 172.21.15.192:0/2314960284 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0813c8d0 con 0x7fde081420e0 2023-12-15T17:01:07.941 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:07.940+0000 7fddfdffb640 1 -- 172.21.15.192:0/2314960284 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 2) v1 ==== 44706+0+0 (secure 0 0 0) 0x7fde00012760 con 0x7fde081420e0 2023-12-15T17:01:08.050 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.048+0000 7fddfdffb640 1 -- 172.21.15.192:0/2314960284 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fde00011560 con 0x7fde081420e0 2023-12-15T17:01:08.050 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.048+0000 7fddfdffb640 10 client.4160.objecter ms_dispatch 0x7fde08000b90 osd_map(9..9 src has 1..9) v4 2023-12-15T17:01:08.050 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.048+0000 7fddfdffb640 3 client.4160.objecter handle_osd_map got epochs [9,9] > 0 2023-12-15T17:01:08.050 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.048+0000 7fddfdffb640 3 client.4160.objecter handle_osd_map decoding full epoch 9 2023-12-15T17:01:08.050 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.048+0000 7fddfdffb640 20 client.4160.objecter dump_active .. 0 homeless 2023-12-15T17:01:08.050 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.048+0000 7fddfdffb640 1 -- 172.21.15.192:0/2314960284 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7fde00007070 con 0x7fde081420e0 2023-12-15T17:01:08.051 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.048+0000 7fddfdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:08.051 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.048+0000 7fddfdffb640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:08.118 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.116+0000 7fde0f533640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ac6e8a35-6df7-48d4-b59a-1f87eceaacfa", "id": 7}] 2023-12-15T17:01:08.118 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.116+0000 7fde0f533640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ac6e8a35-6df7-48d4-b59a-1f87eceaacfa", "id": 7}] 2023-12-15T17:01:08.118 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.116+0000 7fde0f533640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:08.118 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.116+0000 7fde0f533640 1 -- 172.21.15.192:0/2314960284 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ac6e8a35-6df7-48d4-b59a-1f87eceaacfa", "id": 7} v 0) v1 -- 0x7fde08075b60 con 0x7fde081420e0 2023-12-15T17:01:08.477 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.476+0000 7fddfdffb640 1 -- 172.21.15.192:0/2314960284 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ac6e8a35-6df7-48d4-b59a-1f87eceaacfa", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fde00017070 con 0x7fde081420e0 2023-12-15T17:01:08.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.476+0000 7fddfdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ac6e8a35-6df7-48d4-b59a-1f87eceaacfa", "id": 7}] 2023-12-15T17:01:08.478 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.476+0000 7fddfdffb640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:08.478 INFO:teuthology.orchestra.run.smithi192.stdout:7 2023-12-15T17:01:08.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 10 librados: watch_flush enter 2023-12-15T17:01:08.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 10 librados: watch_flush exit 2023-12-15T17:01:08.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 20 client.4160.objecter shutdown clearing up homeless session... 2023-12-15T17:01:08.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 10 client.4160.objecter successfully canceled tick 2023-12-15T17:01:08.479 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 10 monclient: shutdown 2023-12-15T17:01:08.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:08.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 1 -- 172.21.15.192:0/2314960284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde081420e0 msgr2=0x7fde08131d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:08.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 1 --2- 172.21.15.192:0/2314960284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde081420e0 0x7fde08131d60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde0000caa0 tx=0x7fde0000cf60 comp rx=0 tx=0).stop 2023-12-15T17:01:08.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 1 -- 172.21.15.192:0/2314960284 shutdown_connections 2023-12-15T17:01:08.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 1 --2- 172.21.15.192:0/2314960284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde081420e0 0x7fde08131d60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:08.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 1 --2- 172.21.15.192:0/2314960284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde080a8d30 0x7fde08131820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:08.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 1 --2- 172.21.15.192:0/2314960284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde080a8950 0x7fde0813a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:08.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 1 -- 172.21.15.192:0/2314960284 >> 172.21.15.192:0/2314960284 conn(0x7fde080b77d0 msgr2=0x7fde08134f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:08.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 1 -- 172.21.15.192:0/2314960284 shutdown_connections 2023-12-15T17:01:08.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 1 -- 172.21.15.192:0/2314960284 wait complete. 2023-12-15T17:01:08.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-12-15T17:01:08.477+0000 7fde0f533640 1 librados: shutdown 2023-12-15T17:01:08.488 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-15T17:01:08.489 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-15T17:01:08.489 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-12-15T17:01:08.491 INFO:tasks.ceph.osd.0:Started 2023-12-15T17:01:08.491 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-15T17:01:08.491 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-15T17:01:08.491 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-12-15T17:01:08.493 INFO:tasks.ceph.osd.1:Started 2023-12-15T17:01:08.493 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-15T17:01:08.494 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-15T17:01:08.494 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-12-15T17:01:08.496 INFO:tasks.ceph.osd.2:Started 2023-12-15T17:01:08.496 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-15T17:01:08.496 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-15T17:01:08.496 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-12-15T17:01:08.498 INFO:tasks.ceph.osd.3:Started 2023-12-15T17:01:08.498 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-15T17:01:08.499 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-15T17:01:08.499 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-12-15T17:01:08.501 INFO:tasks.ceph.osd.4:Started 2023-12-15T17:01:08.501 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-15T17:01:08.502 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-15T17:01:08.502 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-12-15T17:01:08.504 INFO:tasks.ceph.osd.5:Started 2023-12-15T17:01:08.504 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-15T17:01:08.504 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-15T17:01:08.505 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-12-15T17:01:08.506 INFO:tasks.ceph.osd.6:Started 2023-12-15T17:01:08.506 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-15T17:01:08.506 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-15T17:01:08.507 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-12-15T17:01:08.509 INFO:tasks.ceph.osd.7:Started 2023-12-15T17:01:08.509 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-15T17:01:08.653 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.652+0000 7f20993e9640 10 monclient: get_monmap_and_config 2023-12-15T17:01:08.653 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.652+0000 7f20993e9640 10 monclient: build_initial_monmap 2023-12-15T17:01:08.653 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.652+0000 7f20993e9640 10 monclient: monmap: 2023-12-15T17:01:08.653 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:01:08.653 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:08.654 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:01:08.653036+0000 2023-12-15T17:01:08.654 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:01:08.653036+0000 2023-12-15T17:01:08.654 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:08.654 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:08.654 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:08.654 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:08.654 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:08.654 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:08.654 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:08.654 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 1 Processor -- start 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 1 -- start start 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 10 monclient: init 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20940a8910 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f20940a9b70 0x7f209413f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 10 monclient(hunting): picked mon.noname-a con 0x7f20940a9b70 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2094141f90 0x7f20941423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 10 monclient(hunting): picked mon.noname-b con 0x7f2094141f90 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20940a8cf0 con 0x7f20940a8910 2023-12-15T17:01:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.653+0000 7f20993e9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.654+0000 7f20993e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20940aabd0 con 0x7f20940a9b70 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.654+0000 7f20993e9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.654+0000 7f20993e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20940aaeb0 con 0x7f2094141f90 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.654+0000 7f20993e9640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.654+0000 7f20993e9640 10 monclient(hunting): authenticate will time out at 1372.969436s 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.654+0000 7f20927fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f20940a9b70 0x7f209413f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.654+0000 7f20937fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2094141f90 0x7f20941423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20937fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2094141f90 0x7f20941423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:50332/0 (socket says 172.21.15.69:50332) 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20937fe640 1 -- 172.21.15.69:0/363047594 learned_addr learned my addr 172.21.15.69:0/363047594 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20937fe640 10 monclient(hunting): get_auth_request con 0x7f2094141f90 auth_method 0 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20937fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20937fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20927fc640 10 monclient(hunting): get_auth_request con 0x7f20940a9b70 auth_method 0 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20927fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:08.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20927fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:08.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:08.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:08.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:08.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:08.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:08.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.655+0000 7f20927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.657+0000 7f20937fe640 10 monclient(hunting): handle_auth_done global_id 4163 payload 274 2023-12-15T17:01:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.657+0000 7f20937fe640 1 -- 172.21.15.69:0/363047594 >> v2:172.21.15.69:3300/0 conn(0x7f20940a9b70 msgr2=0x7f209413f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.657+0000 7f20937fe640 1 --2- 172.21.15.69:0/363047594 >> v2:172.21.15.69:3300/0 conn(0x7f20940a9b70 0x7f209413f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.657+0000 7f20937fe640 1 -- 172.21.15.69:0/363047594 >> v1:172.21.15.192:6789/0 conn(0x7f20940a8910 legacy=0x7f209413ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.658+0000 7f20937fe640 10 monclient: _finish_hunting 0 2023-12-15T17:01:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.658+0000 7f20937fe640 1 monclient: found mon.noname-b 2023-12-15T17:01:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.658+0000 7f20937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:08.661 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.658+0000 7f20937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:01:08.661 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.658+0000 7f20937fe640 1 -- 172.21.15.69:0/363047594 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20940a9720 con 0x7f2094141f90 2023-12-15T17:01:08.661 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.658+0000 7f20927fc640 1 --2- 172.21.15.69:0/363047594 >> v2:172.21.15.69:3300/0 conn(0x7f20940a9b70 0x7f209413f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-15T17:01:08.661 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.660+0000 7f20937fe640 1 --2- 172.21.15.69:0/363047594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2094141f90 0x7f20941423f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2094074590 tx=0x7f208c02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d7b5300211f9f8a server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 1 -- 172.21.15.69:0/363047594 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c02ca50 con 0x7f2094141f90 2023-12-15T17:01:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 10 monclient: dump: 2023-12-15T17:01:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 10 monclient: _finish_auth 0 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 10 monclient: _check_auth_tickets 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 1 -- 172.21.15.69:0/363047594 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208c02cbb0 con 0x7f2094141f90 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 1 -- 172.21.15.69:0/363047594 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c0343f0 con 0x7f2094141f90 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f2091ffb640 10 monclient: dump: 2023-12-15T17:01:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f20993e9640 5 monclient: authenticate success, global_id 4163 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f20993e9640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.665+0000 7f20993e9640 4 set_mon_vals no callback set 2023-12-15T17:01:08.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.666+0000 7f20993e9640 10 monclient: shutdown 2023-12-15T17:01:08.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.666+0000 7f20993e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:08.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.666+0000 7f20993e9640 1 -- 172.21.15.69:0/363047594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2094141f90 msgr2=0x7f20941423f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:08.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.666+0000 7f20993e9640 1 --2- 172.21.15.69:0/363047594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2094141f90 0x7f20941423f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2094074590 tx=0x7f208c02beb0 comp rx=0 tx=0).stop 2023-12-15T17:01:08.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.666+0000 7f20993e9640 1 -- 172.21.15.69:0/363047594 shutdown_connections 2023-12-15T17:01:08.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.666+0000 7f20993e9640 1 --2- 172.21.15.69:0/363047594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2094141f90 0x7f20941423f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.666+0000 7f20993e9640 1 --2- 172.21.15.69:0/363047594 >> v2:172.21.15.69:3300/0 conn(0x7f20940a9b70 0x7f209413f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.666+0000 7f20993e9640 1 -- 172.21.15.69:0/363047594 >> 172.21.15.69:0/363047594 conn(0x7f20940b7780 msgr2=0x7f20940b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.666+0000 7f20993e9640 1 -- 172.21.15.69:0/363047594 shutdown_connections 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.666+0000 7f20993e9640 1 -- 172.21.15.69:0/363047594 wait complete. 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.667+0000 7f20993e9640 10 monclient: build_initial_monmap 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.667+0000 7f20993e9640 10 monclient: monmap: 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.667+0000 7f20993e9640 1 librados: starting msgr at 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.667+0000 7f20993e9640 1 librados: starting objecter 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 1 Processor -- start 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 1 -- start start 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 1 librados: setting wanted keys 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 1 librados: calling monclient init 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 10 monclient: init 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20940a8910 0x7f20941376b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 10 monclient(hunting): picked mon.noname-c con 0x7f20940a8910 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2094141f90 0x7f20941359a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 10 monclient(hunting): picked mon.noname-a con 0x7f2094141f90 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f209413ae50 0x7f2094138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 10 monclient(hunting): picked mon.noname-b con 0x7f209413ae50 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20940ac100 con 0x7f2094141f90 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20940ac370 con 0x7f209413ae50 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209413b2c0 con 0x7f20940a8910 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.668+0000 7f20993e9640 10 monclient(hunting): authenticate will time out at 1372.984339s 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.669+0000 7f20927fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2094141f90 0x7f20941359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.669+0000 7f2092ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20940a8910 0x7f20941376b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f20927fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2094141f90 0x7f20941359a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54576/0 (socket says 172.21.15.69:54576) 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f20927fc640 1 -- 172.21.15.69:0/1986085371 learned_addr learned my addr 172.21.15.69:0/1986085371 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f20927fc640 10 monclient(hunting): get_auth_request con 0x7f2094141f90 auth_method 0 2023-12-15T17:01:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f20927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f20927fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f20927fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f2092ffd640 10 monclient(hunting): get_auth_request con 0x7f20940a8910 auth_method 0 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f2092ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f2092ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f2092ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f20927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f20927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.670+0000 7f20927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.671+0000 7f2092ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.671+0000 7f2092ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.671+0000 7f2092ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.673+0000 7f2092ffd640 10 monclient(hunting): handle_auth_done global_id 4166 payload 824 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.673+0000 7f2092ffd640 1 -- 172.21.15.69:0/1986085371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f209413ae50 msgr2=0x7f2094138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.673+0000 7f2092ffd640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f209413ae50 0x7f2094138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.673+0000 7f2092ffd640 1 -- 172.21.15.69:0/1986085371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2094141f90 msgr2=0x7f20941359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.673+0000 7f2092ffd640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2094141f90 0x7f20941359a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.673+0000 7f2092ffd640 10 monclient: _finish_hunting 0 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.673+0000 7f2092ffd640 1 monclient: found mon.noname-c 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.673+0000 7f2092ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.673+0000 7f2092ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.673+0000 7f2092ffd640 1 -- 172.21.15.69:0/1986085371 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2088009400 con 0x7f20940a8910 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.676+0000 7f2092ffd640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20940a8910 0x7f20941376b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f208400c410 tx=0x7f208400c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.676+0000 7f207b7fe640 10 client.?.objecter ms_handle_connect 0x7f20940a8910 2023-12-15T17:01:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.676+0000 7f207b7fe640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:08.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f207b7fe640 1 -- 172.21.15.69:0/1986085371 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20840035f0 con 0x7f20940a8910 2023-12-15T17:01:08.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f207b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:08.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f207b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:01:08.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f207b7fe640 10 monclient: dump: 2023-12-15T17:01:08.681 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:08.681 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:08.681 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f207b7fe640 10 monclient: _finish_auth 0 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f207b7fe640 10 monclient: _check_auth_tickets 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f207b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f20993e9640 5 monclient: authenticate success, global_id 4166 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f20993e9640 10 monclient: _renew_subs 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f20993e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f20993e9640 1 -- 172.21.15.69:0/1986085371 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c009b30 con 0x7f20940a8910 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f20993e9640 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:08.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f20993e9640 10 monclient: _renew_subs 2023-12-15T17:01:08.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f20993e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:08.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f20993e9640 1 -- 172.21.15.69:0/1986085371 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2094138920 con 0x7f20940a8910 2023-12-15T17:01:08.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.679+0000 7f20993e9640 1 librados: init done 2023-12-15T17:01:08.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.680+0000 7f20993e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:08.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.680+0000 7f20993e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:08.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.680+0000 7f20993e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:08.683 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.680+0000 7f20993e9640 1 -- 172.21.15.69:0/1986085371 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2094075b60 con 0x7f20940a8910 2023-12-15T17:01:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.681+0000 7f207b7fe640 1 -- 172.21.15.69:0/1986085371 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2084003750 con 0x7f20940a8910 2023-12-15T17:01:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.681+0000 7f207b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.681+0000 7f207b7fe640 1 -- 172.21.15.69:0/1986085371 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084003b40 con 0x7f20940a8910 2023-12-15T17:01:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.681+0000 7f207b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.681+0000 7f207b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.681+0000 7f207b7fe640 10 monclient: dump: 2023-12-15T17:01:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:08.685 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.681+0000 7f20917fa640 4 set_mon_vals no callback set 2023-12-15T17:01:08.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.694+0000 7f207b7fe640 1 -- 172.21.15.69:0/1986085371 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 2) v1 ==== 44706+0+0 (secure 0 0 0) 0x7f2084003ca0 con 0x7f20940a8910 2023-12-15T17:01:08.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.871+0000 7f207b7fe640 1 -- 172.21.15.69:0/1986085371 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2084010920 con 0x7f20940a8910 2023-12-15T17:01:08.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.871+0000 7f207b7fe640 10 client.4166.objecter ms_dispatch 0x7f2094000b90 osd_map(10..10 src has 1..10) v4 2023-12-15T17:01:08.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.871+0000 7f207b7fe640 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2023-12-15T17:01:08.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.871+0000 7f207b7fe640 3 client.4166.objecter handle_osd_map decoding full epoch 10 2023-12-15T17:01:08.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.871+0000 7f207b7fe640 20 client.4166.objecter dump_active .. 0 homeless 2023-12-15T17:01:08.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.871+0000 7f207b7fe640 1 -- 172.21.15.69:0/1986085371 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f208400eb00 con 0x7f20940a8910 2023-12-15T17:01:08.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.871+0000 7f207b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:08.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.871+0000 7f207b7fe640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:08.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.993+0000 7f20993e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:08.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.993+0000 7f20993e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:08.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.993+0000 7f20993e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:08.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:08.993+0000 7f20993e9640 1 -- 172.21.15.69:0/1986085371 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f209413e310 con 0x7f20940a8910 2023-12-15T17:01:09.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.502+0000 7f207b7fe640 1 -- 172.21.15.69:0/1986085371 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mgrmap(e 3) v1 ==== 89208+0+0 (secure 0 0 0) 0x7f208400ed30 con 0x7f20940a8910 2023-12-15T17:01:09.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.502+0000 7f207b7fe640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f208006d180 0x7f208006f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:09.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.503+0000 7f20927fc640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f208006d180 0x7f208006f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:09.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.503+0000 7f20927fc640 10 monclient: get_auth_request con 0x7f208006d180 auth_method 0 2023-12-15T17:01:09.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.503+0000 7f207b7fe640 10 client.4166.objecter ms_handle_connect 0x7f208006d180 2023-12-15T17:01:09.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.503+0000 7f20927fc640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f208006d180 0x7f208006f570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f209414c590 tx=0x7f2088009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:09.505 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:09.504+0000 7f7051b9c640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-15T17:01:09.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.507+0000 7f207b7fe640 1 -- 172.21.15.69:0/1986085371 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f2084034680 con 0x7f20940a8910 2023-12-15T17:01:09.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.507+0000 7f207b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:09.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.507+0000 7f207b7fe640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:09.508 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:01:09.508 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":10,"fsid":"1a9f4b8c-183d-4435-93ef-5057f2b4aded","created":"2023-12-15T17:01:02.006221+0000","modified":"2023-12-15T17:01:08.340674+0000","last_up_change":"0.000000","last_in_change":"2023-12-15T17:01:08.340674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51cce62a-d177-4e4b-a814-7e537407f397","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1556db3a-b386-444f-8327-3dce2ad46d33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eaac40b5-9fc0-4e13-a322-763a9f3b685f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"71221e7e-8713-4fc4-bef1-c1cbe40627da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2c9feecc-d3b5-429d-98fe-e6eddc91eb8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac6e8a35-6df7-48d4-b59a-1f87eceaacfa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 10 librados: watch_flush enter 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 10 librados: watch_flush exit 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 20 client.4166.objecter shutdown clearing up homeless session... 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 10 client.4166.objecter successfully canceled tick 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 1 -- 172.21.15.69:0/1986085371 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f208006d180 msgr2=0x7f208006f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f208006d180 0x7f208006f570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f209414c590 tx=0x7f2088009190 comp rx=0 tx=0).stop 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 10 monclient: shutdown 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 1 -- 172.21.15.69:0/1986085371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20940a8910 msgr2=0x7f20941376b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20940a8910 0x7f20941376b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f208400c410 tx=0x7f208400c810 comp rx=0 tx=0).stop 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 1 -- 172.21.15.69:0/1986085371 shutdown_connections 2023-12-15T17:01:09.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f208006d180 0x7f208006f570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:09.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f209413ae50 0x7f2094138200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:09.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2094141f90 0x7f20941359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:09.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 1 --2- 172.21.15.69:0/1986085371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20940a8910 0x7f20941376b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:09.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.508+0000 7f20993e9640 1 -- 172.21.15.69:0/1986085371 >> 172.21.15.69:0/1986085371 conn(0x7f20940b7780 msgr2=0x7f209413f180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:09.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.509+0000 7f20993e9640 1 -- 172.21.15.69:0/1986085371 shutdown_connections 2023-12-15T17:01:09.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.509+0000 7f20993e9640 1 -- 172.21.15.69:0/1986085371 wait complete. 2023-12-15T17:01:09.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:09.509+0000 7f20993e9640 1 librados: shutdown 2023-12-15T17:01:09.526 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-15T17:01:09.526 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-15T17:01:09.600 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:01:09.599+0000 7f7051b9c640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-15T17:01:10.332 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-15T17:01:10.477 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.475+0000 7fbf07075640 10 monclient: get_monmap_and_config 2023-12-15T17:01:10.477 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.475+0000 7fbf07075640 10 monclient: build_initial_monmap 2023-12-15T17:01:10.477 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.475+0000 7fbf07075640 10 monclient: monmap: 2023-12-15T17:01:10.477 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:10.477 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:10.477 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:01:10.475564+0000 2023-12-15T17:01:10.477 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:01:10.475564+0000 2023-12-15T17:01:10.478 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:10.478 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:10.478 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:10.478 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:10.478 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:10.478 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:10.478 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:10.478 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:10.478 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.476+0000 7fbf07075640 1 Processor -- start 2023-12-15T17:01:10.478 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.476+0000 7fbf07075640 1 -- start start 2023-12-15T17:01:10.479 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.476+0000 7fbf07075640 10 monclient: init 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.477+0000 7fbf07075640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.477+0000 7fbf07075640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.477+0000 7fbf07075640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf000a8a00 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.477+0000 7fbf07075640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf00146150 0x7fbf000a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.477+0000 7fbf07075640 10 monclient(hunting): picked mon.noname-b con 0x7fbf00146150 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.477+0000 7fbf07075640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbf0013eca0 0x7fbf0013f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.477+0000 7fbf07075640 10 monclient(hunting): picked mon.noname-a con 0x7fbf0013eca0 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.477+0000 7fbf07075640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf001455b0 con 0x7fbf000a8a00 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.477+0000 7fbf07075640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.478+0000 7fbf07075640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf000aa970 con 0x7fbf0013eca0 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.478+0000 7fbf07075640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.478+0000 7fbf07075640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000aac50 con 0x7fbf00146150 2023-12-15T17:01:10.482 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.478+0000 7fbf07075640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.478+0000 7fbf07075640 10 monclient(hunting): authenticate will time out at 1374.793627s 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.478+0000 7fbf04e11640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fbf000a8a00 0x7fbf00145a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:45312/0 (socket says 172.21.15.69:45312) 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.478+0000 7fbf04e11640 1 -- 172.21.15.69:0/2546310174 learned_addr learned my addr 172.21.15.69:0/2546310174 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.478+0000 7fbf05612640 1 --2- 172.21.15.69:0/2546310174 >> v2:172.21.15.69:3300/0 conn(0x7fbf0013eca0 0x7fbf0013f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbefffff640 1 --2- 172.21.15.69:0/2546310174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf00146150 0x7fbf000a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbefffff640 10 monclient(hunting): get_auth_request con 0x7fbf00146150 auth_method 0 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbefffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbefffff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbefffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbf05612640 10 monclient(hunting): get_auth_request con 0x7fbf0013eca0 auth_method 0 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbf05612640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbf05612640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbf05612640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbf05612640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbf05612640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:10.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.479+0000 7fbf05612640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:10.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.480+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:10.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.480+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:10.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.480+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.481+0000 7fbeff7fe640 1 -- 172.21.15.69:0/2546310174 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293298296 0 0) 0x7fbf001455b0 con 0x7fbf000a8a00 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbeff7fe640 10 monclient(hunting): _negotiate 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbeff7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbeff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbeff7fe640 10 monclient(hunting): my global_id is 4177 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbeff7fe640 1 -- 172.21.15.69:0/2546310174 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef0003520 con 0x7fbf000a8a00 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbf05612640 10 monclient(hunting): handle_auth_done global_id 4155 payload 274 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbf05612640 1 -- 172.21.15.69:0/2546310174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf00146150 msgr2=0x7fbf000a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbf05612640 1 --2- 172.21.15.69:0/2546310174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf00146150 0x7fbf000a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbf05612640 1 -- 172.21.15.69:0/2546310174 >> v1:172.21.15.192:6789/0 conn(0x7fbf000a8a00 legacy=0x7fbf00145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbf05612640 10 monclient: _finish_hunting 0 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbf05612640 1 monclient: found mon.noname-a 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbf05612640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:10.486 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbf05612640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:01:10.487 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.482+0000 7fbf05612640 1 -- 172.21.15.69:0/2546310174 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000aaf30 con 0x7fbf0013eca0 2023-12-15T17:01:10.487 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.485+0000 7fbf05612640 1 --2- 172.21.15.69:0/2546310174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0013eca0 0x7fbf0013f130 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009d90 tx=0x7fbef802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74987d7fbc4e86d8 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 1 -- 172.21.15.69:0/2546310174 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef802cc80 con 0x7fbf0013eca0 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 10 monclient: dump: 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:10.493 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 10 monclient: _finish_auth 0 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 10 monclient: _check_auth_tickets 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 1 -- 172.21.15.69:0/2546310174 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef802cde0 con 0x7fbf0013eca0 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 1 -- 172.21.15.69:0/2546310174 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8034660 con 0x7fbf0013eca0 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbeff7fe640 10 monclient: dump: 2023-12-15T17:01:10.494 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:10.495 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:10.495 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:10.495 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:10.495 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:10.495 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:10.495 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:10.495 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:10.495 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:10.496 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:10.496 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbf07075640 5 monclient: authenticate success, global_id 4155 2023-12-15T17:01:10.496 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbf07075640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:10.496 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.490+0000 7fbf07075640 4 set_mon_vals no callback set 2023-12-15T17:01:10.496 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.491+0000 7fbf07075640 10 monclient: shutdown 2023-12-15T17:01:10.496 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.491+0000 7fbf07075640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:10.496 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.491+0000 7fbf07075640 1 -- 172.21.15.69:0/2546310174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0013eca0 msgr2=0x7fbf0013f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:10.496 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.491+0000 7fbf07075640 1 --2- 172.21.15.69:0/2546310174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0013eca0 0x7fbf0013f130 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009d90 tx=0x7fbef802bed0 comp rx=0 tx=0).stop 2023-12-15T17:01:10.502 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.492+0000 7fbf07075640 1 -- 172.21.15.69:0/2546310174 shutdown_connections 2023-12-15T17:01:10.502 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.492+0000 7fbf07075640 1 --2- 172.21.15.69:0/2546310174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0013eca0 0x7fbf0013f130 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:10.502 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.492+0000 7fbf07075640 1 --2- 172.21.15.69:0/2546310174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf00146150 0x7fbf000a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:10.502 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.492+0000 7fbf07075640 1 -- 172.21.15.69:0/2546310174 >> 172.21.15.69:0/2546310174 conn(0x7fbf00066e80 msgr2=0x7fbf000b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.492+0000 7fbf07075640 1 -- 172.21.15.69:0/2546310174 shutdown_connections 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.493+0000 7fbf07075640 1 -- 172.21.15.69:0/2546310174 wait complete. 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.494+0000 7fbf07075640 10 monclient: build_initial_monmap 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.494+0000 7fbf07075640 10 monclient: monmap: 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:10.503 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.495+0000 7fbf07075640 1 librados: starting msgr at 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.495+0000 7fbf07075640 1 librados: starting objecter 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.495+0000 7fbf07075640 1 Processor -- start 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.495+0000 7fbf07075640 1 -- start start 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.495+0000 7fbf07075640 1 librados: setting wanted keys 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.495+0000 7fbf07075640 1 librados: calling monclient init 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.495+0000 7fbf07075640 10 monclient: init 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.496+0000 7fbf07075640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.496+0000 7fbf07075640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.496+0000 7fbf07075640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf000a8a00 0x7fbf000ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.496+0000 7fbf07075640 10 monclient(hunting): picked mon.noname-c con 0x7fbf000a8a00 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.496+0000 7fbf07075640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0013eca0 0x7fbf001d7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.496+0000 7fbf07075640 10 monclient(hunting): picked mon.noname-a con 0x7fbf0013eca0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.496+0000 7fbf07075640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00146150 0x7fbf001d75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:10.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.496+0000 7fbf07075640 10 monclient(hunting): picked mon.noname-b con 0x7fbf00146150 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.496+0000 7fbf07075640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.496+0000 7fbf07075640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000acf60 con 0x7fbf0013eca0 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.496+0000 7fbf07075640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.497+0000 7fbf07075640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000aef90 con 0x7fbf00146150 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.497+0000 7fbf07075640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.497+0000 7fbf07075640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf001d49b0 con 0x7fbf000a8a00 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.497+0000 7fbf07075640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.497+0000 7fbf07075640 10 monclient(hunting): authenticate will time out at 1374.812449s 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.497+0000 7fbefffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0013eca0 0x7fbf001d7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.497+0000 7fbf04e11640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf000a8a00 0x7fbf000ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.497+0000 7fbf05612640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00146150 0x7fbf001d75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:10.505 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf04e11640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf000a8a00 0x7fbf000ae510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:50348/0 (socket says 172.21.15.69:50348) 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf04e11640 1 -- 172.21.15.69:0/2505995645 learned_addr learned my addr 172.21.15.69:0/2505995645 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf04e11640 10 monclient(hunting): get_auth_request con 0x7fbf000a8a00 auth_method 0 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf04e11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf04e11640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf04e11640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf05612640 10 monclient(hunting): get_auth_request con 0x7fbf00146150 auth_method 0 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf05612640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf05612640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf05612640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbefffff640 10 monclient(hunting): get_auth_request con 0x7fbf0013eca0 auth_method 0 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbefffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:10.506 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbefffff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:10.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbefffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:10.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:10.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:10.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:10.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf05612640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:10.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf05612640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:10.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.498+0000 7fbf05612640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:10.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.499+0000 7fbf04e11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:10.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.499+0000 7fbf04e11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:10.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.499+0000 7fbf04e11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:10.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.503+0000 7fbf05612640 10 monclient(hunting): handle_auth_done global_id 4180 payload 824 2023-12-15T17:01:10.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.503+0000 7fbf05612640 1 -- 172.21.15.69:0/2505995645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf000a8a00 msgr2=0x7fbf000ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:10.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.503+0000 7fbf05612640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf000a8a00 0x7fbf000ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:10.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.503+0000 7fbf05612640 1 -- 172.21.15.69:0/2505995645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0013eca0 msgr2=0x7fbf001d7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:10.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.503+0000 7fbf05612640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0013eca0 0x7fbf001d7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:10.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.503+0000 7fbf05612640 10 monclient: _finish_hunting 0 2023-12-15T17:01:10.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.503+0000 7fbf05612640 1 monclient: found mon.noname-b 2023-12-15T17:01:10.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.503+0000 7fbf05612640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:10.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.503+0000 7fbf05612640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:01:10.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.503+0000 7fbf05612640 1 -- 172.21.15.69:0/2505995645 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef8009a70 con 0x7fbf00146150 2023-12-15T17:01:10.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.506+0000 7fbf05612640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00146150 0x7fbf001d75d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbef8033000 tx=0x7fbef8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:10.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.506+0000 7fbefd7fa640 10 client.?.objecter ms_handle_connect 0x7fbf00146150 2023-12-15T17:01:10.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.506+0000 7fbefd7fa640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:10.512 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbefd7fa640 1 -- 172.21.15.69:0/2505995645 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8040650 con 0x7fbf00146150 2023-12-15T17:01:10.512 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbefd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:10.512 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbefd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:10.512 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbefd7fa640 10 monclient: dump: 2023-12-15T17:01:10.512 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbefd7fa640 10 monclient: _finish_auth 0 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbefd7fa640 10 monclient: _check_auth_tickets 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbefd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbf07075640 5 monclient: authenticate success, global_id 4180 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbf07075640 10 monclient: _renew_subs 2023-12-15T17:01:10.513 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbf07075640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:10.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbf07075640 1 -- 172.21.15.69:0/2505995645 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf000b2470 con 0x7fbf00146150 2023-12-15T17:01:10.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbf07075640 10 client.4180.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:10.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbf07075640 10 monclient: _renew_subs 2023-12-15T17:01:10.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbf07075640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:10.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.509+0000 7fbf07075640 1 -- 172.21.15.69:0/2505995645 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf001d7b10 con 0x7fbf00146150 2023-12-15T17:01:10.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.510+0000 7fbf07075640 1 librados: init done 2023-12-15T17:01:10.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.511+0000 7fbf07075640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:10.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.511+0000 7fbf07075640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:10.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.511+0000 7fbf07075640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:10.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.511+0000 7fbf07075640 1 -- 172.21.15.69:0/2505995645 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf00069c00 con 0x7fbf00146150 2023-12-15T17:01:10.963 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.962+0000 7fbefd7fa640 1 -- 172.21.15.69:0/2505995645 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef80407b0 con 0x7fbf00146150 2023-12-15T17:01:10.963 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.962+0000 7fbefd7fa640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:10.963 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.962+0000 7fbefd7fa640 1 -- 172.21.15.69:0/2505995645 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8040b60 con 0x7fbf00146150 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.962+0000 7fbefd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.962+0000 7fbefd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.962+0000 7fbefd7fa640 10 monclient: dump: 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:10.964 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.962+0000 7fbefeffd640 4 set_mon_vals no callback set 2023-12-15T17:01:10.965 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.964+0000 7fbefd7fa640 1 -- 172.21.15.69:0/2505995645 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 3) v1 ==== 89208+0+0 (secure 0 0 0) 0x7fbef8052430 con 0x7fbf00146150 2023-12-15T17:01:10.965 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.965+0000 7fbefd7fa640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbee406c9a0 0x7fbee406ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:10.967 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.967+0000 7fbf04e11640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbee406c9a0 0x7fbee406ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:10.967 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.967+0000 7fbf04e11640 10 monclient: get_auth_request con 0x7fbee406c9a0 auth_method 0 2023-12-15T17:01:10.971 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.970+0000 7fbefd7fa640 10 client.4180.objecter ms_handle_connect 0x7fbee406c9a0 2023-12-15T17:01:10.971 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.970+0000 7fbf04e11640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbee406c9a0 0x7fbee406ee60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbef40011e0 tx=0x7fbef4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:10.995 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.994+0000 7fbefd7fa640 1 -- 172.21.15.69:0/2505995645 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fbef807b410 con 0x7fbf00146150 2023-12-15T17:01:10.995 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.994+0000 7fbefd7fa640 10 client.4180.objecter ms_dispatch 0x7fbf00000b90 osd_map(10..10 src has 1..10) v4 2023-12-15T17:01:10.995 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.994+0000 7fbefd7fa640 3 client.4180.objecter handle_osd_map got epochs [10,10] > 0 2023-12-15T17:01:10.995 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.994+0000 7fbefd7fa640 3 client.4180.objecter handle_osd_map decoding full epoch 10 2023-12-15T17:01:10.995 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.994+0000 7fbefd7fa640 20 client.4180.objecter dump_active .. 0 homeless 2023-12-15T17:01:10.995 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.994+0000 7fbefd7fa640 1 -- 172.21.15.69:0/2505995645 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fbef8077750 con 0x7fbf00146150 2023-12-15T17:01:10.995 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.994+0000 7fbefd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:10.995 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:10.994+0000 7fbefd7fa640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:11.345 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.344+0000 7fbefd7fa640 1 -- 172.21.15.69:0/2505995645 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mgrmap(e 4) v1 ==== 89314+0+0 (secure 0 0 0) 0x7fbef8077b20 con 0x7fbf00146150 2023-12-15T17:01:11.356 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.355+0000 7fbf07075640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:11.356 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.355+0000 7fbf07075640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:11.356 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.355+0000 7fbf07075640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:11.356 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.355+0000 7fbf07075640 1 -- 172.21.15.69:0/2505995645 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf0013dae0 con 0x7fbf00146150 2023-12-15T17:01:11.588 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.587+0000 7fbefd7fa640 1 -- 172.21.15.69:0/2505995645 <== mon.1 v2:172.21.15.192:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fbef80a6520 con 0x7fbf00146150 2023-12-15T17:01:11.588 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.587+0000 7fbefd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:11.588 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.587+0000 7fbefd7fa640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:11.588 INFO:teuthology.misc.health.smithi069.stdout: 2023-12-15T17:01:11.588 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"1a9f4b8c-183d-4435-93ef-5057f2b4aded","created":"2023-12-15T17:01:02.006221+0000","modified":"2023-12-15T17:01:08.340674+0000","last_up_change":"0.000000","last_in_change":"2023-12-15T17:01:08.340674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51cce62a-d177-4e4b-a814-7e537407f397","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1556db3a-b386-444f-8327-3dce2ad46d33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eaac40b5-9fc0-4e13-a322-763a9f3b685f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"71221e7e-8713-4fc4-bef1-c1cbe40627da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2c9feecc-d3b5-429d-98fe-e6eddc91eb8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac6e8a35-6df7-48d4-b59a-1f87eceaacfa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T17:01:11.592 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.589+0000 7fbf07075640 10 librados: watch_flush enter 2023-12-15T17:01:11.592 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 10 librados: watch_flush exit 2023-12-15T17:01:11.592 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 20 client.4180.objecter shutdown clearing up homeless session... 2023-12-15T17:01:11.592 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 10 client.4180.objecter successfully canceled tick 2023-12-15T17:01:11.592 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 -- 172.21.15.69:0/2505995645 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbee406c9a0 msgr2=0x7fbee406ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbee406c9a0 0x7fbee406ee60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbef40011e0 tx=0x7fbef4009040 comp rx=0 tx=0).stop 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 10 monclient: shutdown 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 -- 172.21.15.69:0/2505995645 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00146150 msgr2=0x7fbf001d75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00146150 0x7fbf001d75d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbef8033000 tx=0x7fbef8004a40 comp rx=0 tx=0).stop 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 -- 172.21.15.69:0/2505995645 shutdown_connections 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbee406c9a0 0x7fbee406ee60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00146150 0x7fbf001d75d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0013eca0 0x7fbf001d7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 --2- 172.21.15.69:0/2505995645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf000a8a00 0x7fbf000ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 -- 172.21.15.69:0/2505995645 >> 172.21.15.69:0/2505995645 conn(0x7fbf00066e80 msgr2=0x7fbf00144c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 -- 172.21.15.69:0/2505995645 shutdown_connections 2023-12-15T17:01:11.593 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 -- 172.21.15.69:0/2505995645 wait complete. 2023-12-15T17:01:11.594 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:11.590+0000 7fbf07075640 1 librados: shutdown 2023-12-15T17:01:11.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-15T17:01:16.514 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:07.944 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.559 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:07.989 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.584 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:08.014 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.623 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:08.053 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.674 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:08.086 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.734 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:08.145 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.751 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:08.174 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.801 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:08.233 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.816 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:08.239 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.864 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:08.296 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.867 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:08.305 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.926 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:08.363 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.929 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:08.358 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.933 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:08.364 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.991 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:08.420 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:16.996 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:08.427 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:17.142 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:01:17.135+0000 5dc2340 -1 Falling back to public interface 2023-12-15T17:01:17.172 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-12-15T17:01:17.165+0000 5dc2340 -1 Falling back to public interface 2023-12-15T17:01:17.305 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-12-15T17:01:17.298+0000 5dc2340 -1 Falling back to public interface 2023-12-15T17:01:17.365 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:01:17.357+0000 5dc2340 -1 Falling back to public interface 2023-12-15T17:01:17.412 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-12-15T17:01:17.404+0000 5dc2340 -1 Falling back to public interface 2023-12-15T17:01:17.499 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:01:17.492+0000 5dc2340 -1 Falling back to public interface 2023-12-15T17:01:17.555 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:08.985 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:17.560 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:01:17.552+0000 5dc2340 -1 Falling back to public interface 2023-12-15T17:01:17.569 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-12-15T17:01:17.561+0000 5dc2340 -1 Falling back to public interface 2023-12-15T17:01:17.626 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:09.056 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:17.780 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:09.192 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:17.789 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:09.212 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:17.853 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:09.284 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:17.914 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:09.344 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:17.940 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:09.378 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.004 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:09.434 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.006 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:09.435 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.032 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:09.463 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.056 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:09.493 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.139 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:09.550 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.145 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:09.568 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.206 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:09.637 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.249 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:09.660 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.286 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:09.716 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.372 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:09.801 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.382 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:09.812 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.386 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:09.817 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.416 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-15T17:01:18.419 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:09.857 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.503 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.501+0000 7f602c690640 10 monclient: get_monmap_and_config 2023-12-15T17:01:18.503 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.501+0000 7f602c690640 10 monclient: build_initial_monmap 2023-12-15T17:01:18.504 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.501+0000 7f602c690640 10 monclient: monmap: 2023-12-15T17:01:18.504 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:18.504 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:18.504 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:01:18.503335+0000 2023-12-15T17:01:18.504 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:01:18.503335+0000 2023-12-15T17:01:18.504 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:18.504 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:18.504 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:18.504 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:18.505 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:18.505 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:18.505 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:18.505 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:18.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.502+0000 7f602c690640 1 Processor -- start 2023-12-15T17:01:18.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.502+0000 7f602c690640 1 -- start start 2023-12-15T17:01:18.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.502+0000 7f602c690640 10 monclient: init 2023-12-15T17:01:18.507 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:18.508 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-12-15T17:01:18.508 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f60240a89e0 0x7f602413ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:18.508 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 10 monclient(hunting): picked mon.noname-a con 0x7f60240a89e0 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:18.508 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f602413f460 addr v1:172.21.15.69:6789/0 2023-12-15T17:01:18.508 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f60241428e0 0x7f602414c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:18.508 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 10 monclient(hunting): picked mon.noname-c con 0x7f60241428e0 addr v2:172.21.15.192:3300/0 2023-12-15T17:01:18.508 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60240aa770 con 0x7f602413f460 2023-12-15T17:01:18.508 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:18.508 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602413f860 con 0x7f60240a89e0 2023-12-15T17:01:18.508 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:18.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60240aa9c0 con 0x7f60241428e0 2023-12-15T17:01:18.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:18.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602c690640 10 monclient(hunting): authenticate will time out at 1382.819705s 2023-12-15T17:01:18.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.503+0000 7f602a42c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f60240a89e0 0x7f602413ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:18.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.504+0000 7f6029c2b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f602413f460 0x7f60241421d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55502/0 (socket says 172.21.15.69:55502) 2023-12-15T17:01:18.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.504+0000 7f6029c2b640 1 -- 172.21.15.69:0/2275572398 learned_addr learned my addr 172.21.15.69:0/2275572398 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:01:18.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.504+0000 7f602ac2d640 1 --2- 172.21.15.69:0/2275572398 >> v2:172.21.15.192:3300/0 conn(0x7f60241428e0 0x7f602414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:18.509 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.504+0000 7f602a42c640 10 monclient(hunting): get_auth_request con 0x7f60240a89e0 auth_method 0 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.504+0000 7f602a42c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.504+0000 7f602a42c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.504+0000 7f602a42c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.504+0000 7f602ac2d640 10 monclient(hunting): get_auth_request con 0x7f60241428e0 auth_method 0 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.504+0000 7f602ac2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.504+0000 7f602ac2d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.504+0000 7f602ac2d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.505+0000 7f602a42c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.505+0000 7f602a42c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.505+0000 7f602a42c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.505+0000 7f602ac2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.505+0000 7f602ac2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:18.510 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.505+0000 7f602ac2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:18.511 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.506+0000 7f602a42c640 10 monclient(hunting): handle_auth_done global_id 4194 payload 274 2023-12-15T17:01:18.511 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.506+0000 7f602a42c640 1 -- 172.21.15.69:0/2275572398 >> v2:172.21.15.192:3300/0 conn(0x7f60241428e0 msgr2=0x7f602414c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:18.511 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.506+0000 7f602a42c640 1 --2- 172.21.15.69:0/2275572398 >> v2:172.21.15.192:3300/0 conn(0x7f60241428e0 0x7f602414c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:18.511 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.506+0000 7f602a42c640 1 -- 172.21.15.69:0/2275572398 >> v1:172.21.15.69:6789/0 conn(0x7f602413f460 legacy=0x7f60241421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:18.511 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.506+0000 7f602a42c640 10 monclient: _finish_hunting 0 2023-12-15T17:01:18.512 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.506+0000 7f602a42c640 1 monclient: found mon.noname-a 2023-12-15T17:01:18.512 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.506+0000 7f602a42c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:18.512 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.506+0000 7f602a42c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:01:18.512 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.506+0000 7f602a42c640 1 -- 172.21.15.69:0/2275572398 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60240aaca0 con 0x7f60240a89e0 2023-12-15T17:01:18.512 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.509+0000 7f602a42c640 1 --2- 172.21.15.69:0/2275572398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60240a89e0 0x7f602413ef20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f60240a84a0 tx=0x7f601c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d544348d0c9ec301 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:18.512 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:09.934 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 1 -- 172.21.15.69:0/2275572398 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c02e070 con 0x7f60240a89e0 2023-12-15T17:01:18.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:18.514 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:01:18.515 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 10 monclient: dump: 2023-12-15T17:01:18.515 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:18.515 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:18.515 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:18.515 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:18.515 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:18.515 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:18.515 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:18.515 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:18.516 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:18.516 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:18.516 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 10 monclient: _finish_auth 0 2023-12-15T17:01:18.516 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 10 monclient: _check_auth_tickets 2023-12-15T17:01:18.516 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:18.516 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 1 -- 172.21.15.69:0/2275572398 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c0315a0 con 0x7f60240a89e0 2023-12-15T17:01:18.516 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602c690640 5 monclient: authenticate success, global_id 4194 2023-12-15T17:01:18.516 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602c690640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:18.516 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:18.516 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 1 -- 172.21.15.69:0/2275572398 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c031930 con 0x7f60240a89e0 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602942a640 10 monclient: dump: 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:18.517 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:18.518 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:18.518 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602c690640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:18.518 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.512+0000 7f602c690640 4 set_mon_vals no callback set 2023-12-15T17:01:18.523 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.513+0000 7f602c690640 10 monclient: shutdown 2023-12-15T17:01:18.523 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.513+0000 7f602c690640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:18.523 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.513+0000 7f602c690640 1 -- 172.21.15.69:0/2275572398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60240a89e0 msgr2=0x7f602413ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:18.523 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.513+0000 7f602c690640 1 --2- 172.21.15.69:0/2275572398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60240a89e0 0x7f602413ef20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f60240a84a0 tx=0x7f601c02b840 comp rx=0 tx=0).stop 2023-12-15T17:01:18.523 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.514+0000 7f602c690640 1 -- 172.21.15.69:0/2275572398 shutdown_connections 2023-12-15T17:01:18.523 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.514+0000 7f602c690640 1 --2- 172.21.15.69:0/2275572398 >> v2:172.21.15.192:3300/0 conn(0x7f60241428e0 0x7f602414c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:18.523 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.514+0000 7f602c690640 1 --2- 172.21.15.69:0/2275572398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60240a89e0 0x7f602413ef20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:18.523 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.514+0000 7f602c690640 1 -- 172.21.15.69:0/2275572398 >> 172.21.15.69:0/2275572398 conn(0x7f60240b7940 msgr2=0x7f60240b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:18.523 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.514+0000 7f602c690640 1 -- 172.21.15.69:0/2275572398 shutdown_connections 2023-12-15T17:01:18.523 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.514+0000 7f602c690640 1 -- 172.21.15.69:0/2275572398 wait complete. 2023-12-15T17:01:18.524 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.514+0000 7f602c690640 10 monclient: build_initial_monmap 2023-12-15T17:01:18.524 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.514+0000 7f602c690640 10 monclient: monmap: 2023-12-15T17:01:18.524 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:18.524 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:18.524 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:18.524 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-12-15T17:01:18.524 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:18.524 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:18.524 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:18.524 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:18.524 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:18.525 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:18.525 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.515+0000 7f602c690640 1 librados: starting msgr at 2023-12-15T17:01:18.525 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.515+0000 7f602c690640 1 librados: starting objecter 2023-12-15T17:01:18.525 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.515+0000 7f602c690640 1 Processor -- start 2023-12-15T17:01:18.525 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.515+0000 7f602c690640 1 -- start start 2023-12-15T17:01:18.525 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.515+0000 7f602c690640 1 librados: setting wanted keys 2023-12-15T17:01:18.526 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.515+0000 7f602c690640 1 librados: calling monclient init 2023-12-15T17:01:18.526 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.515+0000 7f602c690640 10 monclient: init 2023-12-15T17:01:18.526 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:18.526 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:01:18.526 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60240a89e0 0x7f60240b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:18.526 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 10 monclient(hunting): picked mon.noname-b con 0x7f60240a89e0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:18.526 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f602413f460 0x7f60241452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:18.526 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 10 monclient(hunting): picked mon.noname-c con 0x7f602413f460 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:18.526 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60241428e0 0x7f6024136c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:18.526 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 10 monclient(hunting): picked mon.noname-a con 0x7f60241428e0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:18.527 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:18.527 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6024145120 con 0x7f60241428e0 2023-12-15T17:01:18.527 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:18.527 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60240b74c0 con 0x7f60240a89e0 2023-12-15T17:01:18.527 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:18.527 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6024145820 con 0x7f602413f460 2023-12-15T17:01:18.527 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:18.527 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602c690640 10 monclient(hunting): authenticate will time out at 1382.832839s 2023-12-15T17:01:18.527 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.516+0000 7f602ac2d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60241428e0 0x7f6024136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:18.527 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f6029c2b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f602413f460 0x7f60241452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602ac2d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60241428e0 0x7f6024136c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51056/0 (socket says 172.21.15.69:51056) 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602ac2d640 1 -- 172.21.15.69:0/886612597 learned_addr learned my addr 172.21.15.69:0/886612597 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602ac2d640 10 monclient(hunting): get_auth_request con 0x7f60241428e0 auth_method 0 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602ac2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602ac2d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602ac2d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602a42c640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60240a89e0 0x7f60240b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f6029c2b640 10 monclient(hunting): get_auth_request con 0x7f602413f460 auth_method 0 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f6029c2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f6029c2b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f6029c2b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:18.528 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602a42c640 10 monclient(hunting): get_auth_request con 0x7f60240a89e0 auth_method 0 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602a42c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602a42c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602a42c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602ac2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602ac2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.517+0000 7f602ac2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.518+0000 7f6029c2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.518+0000 7f6029c2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.518+0000 7f6029c2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.518+0000 7f602a42c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.518+0000 7f602a42c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:18.529 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.518+0000 7f602a42c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:18.530 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.520+0000 7f602ac2d640 10 monclient(hunting): handle_auth_done global_id 4197 payload 824 2023-12-15T17:01:18.530 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.520+0000 7f602ac2d640 1 -- 172.21.15.69:0/886612597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f602413f460 msgr2=0x7f60241452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:18.530 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.520+0000 7f602ac2d640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f602413f460 0x7f60241452e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:18.530 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.520+0000 7f602ac2d640 1 -- 172.21.15.69:0/886612597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60240a89e0 msgr2=0x7f60240b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:18.530 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.520+0000 7f602ac2d640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60240a89e0 0x7f60240b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:18.530 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.520+0000 7f602ac2d640 10 monclient: _finish_hunting 0 2023-12-15T17:01:18.530 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.520+0000 7f602ac2d640 1 monclient: found mon.noname-a 2023-12-15T17:01:18.530 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.520+0000 7f602ac2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:18.530 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.520+0000 7f602ac2d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:01:18.530 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.520+0000 7f602ac2d640 1 -- 172.21.15.69:0/886612597 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601c009400 con 0x7f60241428e0 2023-12-15T17:01:18.530 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.520+0000 7f602a42c640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60240a89e0 0x7f60240b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-15T17:01:18.532 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.522+0000 7f602ac2d640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60241428e0 0x7f6024136c80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f602000efc0 tx=0x7f602000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:18.532 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.522+0000 7f6016ffd640 10 client.?.objecter ms_handle_connect 0x7f60241428e0 2023-12-15T17:01:18.532 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.522+0000 7f6016ffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:18.532 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 1 -- 172.21.15.69:0/886612597 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020014480 con 0x7f60241428e0 2023-12-15T17:01:18.532 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:18.533 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:01:18.533 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 10 monclient: dump: 2023-12-15T17:01:18.533 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:18.533 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:18.533 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:18.533 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:18.533 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:18.533 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:18.533 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:18.533 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:18.533 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 10 monclient: _finish_auth 0 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 1 -- 172.21.15.69:0/886612597 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602000f040 con 0x7f60241428e0 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 1 -- 172.21.15.69:0/886612597 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020014820 con 0x7f60241428e0 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f602c690640 5 monclient: authenticate success, global_id 4197 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f602c690640 10 monclient: _renew_subs 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f602c690640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f602c690640 1 -- 172.21.15.69:0/886612597 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60241371c0 con 0x7f60241428e0 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6028c29640 4 set_mon_vals no callback set 2023-12-15T17:01:18.534 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f6016ffd640 10 monclient: dump: 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:18.535 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f602c690640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:18.536 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f602c690640 10 monclient: _renew_subs 2023-12-15T17:01:18.536 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f602c690640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:18.536 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f602c690640 1 -- 172.21.15.69:0/886612597 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6024137680 con 0x7f60241428e0 2023-12-15T17:01:18.536 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.526+0000 7f602c690640 1 librados: init done 2023-12-15T17:01:18.536 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.527+0000 7f602c690640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:18.536 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.527+0000 7f602c690640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:18.536 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.527+0000 7f602c690640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:18.536 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.527+0000 7f602c690640 1 -- 172.21.15.69:0/886612597 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6024145ec0 con 0x7f60241428e0 2023-12-15T17:01:18.540 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:09.952 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.548 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.546+0000 7f6016ffd640 1 -- 172.21.15.69:0/886612597 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) v1 ==== 89314+0+0 (secure 0 0 0) 0x7f6020014980 con 0x7f60241428e0 2023-12-15T17:01:18.548 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.546+0000 7f6016ffd640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f600806cb10 0x7f600806efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:18.549 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.548+0000 7f602a42c640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f600806cb10 0x7f600806efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:18.550 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.548+0000 7f602a42c640 10 monclient: get_auth_request con 0x7f600806cb10 auth_method 0 2023-12-15T17:01:18.552 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.551+0000 7f6016ffd640 10 client.4197.objecter ms_handle_connect 0x7f600806cb10 2023-12-15T17:01:18.552 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.551+0000 7f602a42c640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f600806cb10 0x7f600806efd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f601c036000 tx=0x7f601c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:18.568 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.000 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.580 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:10.010 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.670 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:10.101 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.717 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.154 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.731 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:10.160 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.767 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.197 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.807 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.805+0000 7f6016ffd640 1 -- 172.21.15.69:0/886612597 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6020059520 con 0x7f60241428e0 2023-12-15T17:01:18.807 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.805+0000 7f6016ffd640 10 client.4197.objecter ms_dispatch 0x7f6024000b90 osd_map(10..10 src has 1..10) v4 2023-12-15T17:01:18.807 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.805+0000 7f6016ffd640 3 client.4197.objecter handle_osd_map got epochs [10,10] > 0 2023-12-15T17:01:18.807 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.805+0000 7f6016ffd640 3 client.4197.objecter handle_osd_map decoding full epoch 10 2023-12-15T17:01:18.807 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.805+0000 7f6016ffd640 20 client.4197.objecter dump_active .. 0 homeless 2023-12-15T17:01:18.807 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.805+0000 7f6016ffd640 1 -- 172.21.15.69:0/886612597 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6020055860 con 0x7f60241428e0 2023-12-15T17:01:18.807 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.805+0000 7f6016ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:18.807 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.805+0000 7f6016ffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:18.809 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.231 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.828 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:10.240 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.862 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.293 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.880 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:10.310 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.966 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:10.396 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:18.981 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.979+0000 7f602c690640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:18.981 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.979+0000 7f602c690640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:18.981 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.979+0000 7f602c690640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:18.981 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:18.979+0000 7f602c690640 1 -- 172.21.15.69:0/886612597 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6024145ec0 con 0x7f60241428e0 2023-12-15T17:01:19.012 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.449 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.023 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:10.450 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.025 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.023+0000 7f6016ffd640 1 -- 172.21.15.69:0/886612597 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f6024145ec0 con 0x7f60241428e0 2023-12-15T17:01:19.026 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.023+0000 7f6016ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:19.026 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.023+0000 7f6016ffd640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:19.026 INFO:teuthology.misc.health.smithi069.stdout: 2023-12-15T17:01:19.026 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"1a9f4b8c-183d-4435-93ef-5057f2b4aded","created":"2023-12-15T17:01:02.006221+0000","modified":"2023-12-15T17:01:08.340674+0000","last_up_change":"0.000000","last_in_change":"2023-12-15T17:01:08.340674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51cce62a-d177-4e4b-a814-7e537407f397","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1556db3a-b386-444f-8327-3dce2ad46d33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eaac40b5-9fc0-4e13-a322-763a9f3b685f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"71221e7e-8713-4fc4-bef1-c1cbe40627da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2c9feecc-d3b5-429d-98fe-e6eddc91eb8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac6e8a35-6df7-48d4-b59a-1f87eceaacfa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T17:01:19.027 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 10 librados: watch_flush enter 2023-12-15T17:01:19.027 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 10 librados: watch_flush exit 2023-12-15T17:01:19.027 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 20 client.4197.objecter shutdown clearing up homeless session... 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 10 client.4197.objecter successfully canceled tick 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 -- 172.21.15.69:0/886612597 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f600806cb10 msgr2=0x7f600806efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f600806cb10 0x7f600806efd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f601c036000 tx=0x7f601c03d040 comp rx=0 tx=0).stop 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 10 monclient: shutdown 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 -- 172.21.15.69:0/886612597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60241428e0 msgr2=0x7f6024136c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60241428e0 0x7f6024136c80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f602000efc0 tx=0x7f602000c490 comp rx=0 tx=0).stop 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 -- 172.21.15.69:0/886612597 shutdown_connections 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f600806cb10 0x7f600806efd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60241428e0 0x7f6024136c80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f602413f460 0x7f60241452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 --2- 172.21.15.69:0/886612597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60240a89e0 0x7f60240b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 -- 172.21.15.69:0/886612597 >> 172.21.15.69:0/886612597 conn(0x7f60240b7940 msgr2=0x7f60240b18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 -- 172.21.15.69:0/886612597 shutdown_connections 2023-12-15T17:01:19.028 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 -- 172.21.15.69:0/886612597 wait complete. 2023-12-15T17:01:19.029 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:19.025+0000 7f602c690640 1 librados: shutdown 2023-12-15T17:01:19.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-15T17:01:19.066 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.497 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.109 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.532 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.121 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:10.533 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.149 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.581 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.169 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:10.599 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.258 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:10.688 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.276 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.713 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.287 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:10.715 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.330 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.761 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.378 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.801 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.386 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:10.798 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.413 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.845 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.438 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:10.868 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.529 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:10.959 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.540 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.978 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.553 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:10.982 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.598 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.029 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.651 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:11.062 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.654 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:11.077 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.679 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:11.111 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.704 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:11.134 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.799 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:11.229 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.811 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:11.248 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.825 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:11.253 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.863 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.294 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.916 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:11.327 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.920 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:11.343 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.951 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:11.383 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:19.969 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:11.399 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.066 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:11.496 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.075 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:11.513 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.090 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:11.519 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.128 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.558 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.189 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:11.611 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.216 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:11.648 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.379 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:11.807 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.412 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:11.824 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.425 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.855 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.465 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:11.895 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.467 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:11.879 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.520 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:11.950 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.591 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:12.028 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.604 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:12.034 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.645 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:12.082 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.670 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:12.101 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.708 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:12.130 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.746 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:12.178 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.798 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:12.221 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.800 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:12.232 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.941 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:12.370 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:20.997 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:12.425 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:21.017 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:12.447 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:21.071 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:12.502 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:25.843 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-15T17:01:25.934 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:17.346 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:25.980 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.979+0000 7fa89a0b4640 10 monclient: get_monmap_and_config 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.979+0000 7fa89a0b4640 10 monclient: build_initial_monmap 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.979+0000 7fa89a0b4640 10 monclient: monmap: 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:01:25.980541+0000 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:01:25.980541+0000 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:25.981 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:25.982 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:25.982 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:25.983 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.979+0000 7fa89a0b4640 1 Processor -- start 2023-12-15T17:01:25.983 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.979+0000 7fa89a0b4640 1 -- start start 2023-12-15T17:01:25.983 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.979+0000 7fa89a0b4640 10 monclient: init 2023-12-15T17:01:25.983 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8940a89e0 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa89413f630 0x7fa8941421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 10 monclient(hunting): picked mon.noname-a con 0x7fa89413f630 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa894142710 addr v1:172.21.15.69:6789/0 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8940aa770 con 0x7fa894142710 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8940aa9c0 con 0x7fa8940a89e0 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8940aaca0 con 0x7fa89413f630 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.980+0000 7fa89a0b4640 10 monclient(hunting): authenticate will time out at 1390.297087s 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.981+0000 7fa8937fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa8940a89e0 0x7fa89413ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:34788/0 (socket says 172.21.15.69:34788) 2023-12-15T17:01:25.984 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.981+0000 7fa8937fe640 1 -- 172.21.15.69:0/4063939925 learned_addr learned my addr 172.21.15.69:0/4063939925 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:01:25.986 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.984+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270646737 0 0) 0x7fa8940aa9c0 con 0x7fa8940a89e0 2023-12-15T17:01:25.986 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.984+0000 7fa8927fc640 10 monclient(hunting): _negotiate 2023-12-15T17:01:25.986 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.984+0000 7fa8927fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:25.986 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.984+0000 7fa8927fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:25.986 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.984+0000 7fa8927fc640 10 monclient(hunting): my global_id is 4207 2023-12-15T17:01:25.986 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.984+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87c003520 con 0x7fa8940a89e0 2023-12-15T17:01:25.990 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.989+0000 7fa892ffd640 1 --2- 172.21.15.69:0/4063939925 >> v2:172.21.15.69:3300/0 conn(0x7fa89413f630 0x7fa8941421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:25.991 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.989+0000 7fa892ffd640 10 monclient(hunting): get_auth_request con 0x7fa89413f630 auth_method 0 2023-12-15T17:01:25.991 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.989+0000 7fa892ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:25.991 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.989+0000 7fa892ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:25.991 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.989+0000 7fa892ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:25.995 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1971282109 0 0) 0x7fa87c003520 con 0x7fa8940a89e0 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 >> v2:172.21.15.69:3300/0 conn(0x7fa89413f630 msgr2=0x7fa8941421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 1 --2- 172.21.15.69:0/4063939925 >> v2:172.21.15.69:3300/0 conn(0x7fa89413f630 0x7fa8941421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 >> v1:172.21.15.69:6789/0 conn(0x7fa894142710 legacy=0x7fa89414c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 10 monclient: _finish_hunting 0 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 1 monclient: found mon.noname-c-legacy 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.192:6789/0 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8940aaf80 con 0x7fa8940a89e0 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 10 monclient: _finish_auth 0 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 10 monclient: _check_auth_tickets 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2578675164 0 0) 0x7fa884001670 con 0x7fa8940a89e0 2023-12-15T17:01:25.996 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa89a0b4640 5 monclient: authenticate success, global_id 4207 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa89a0b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 10 monclient: dump: 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:25.997 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] but i'm connected to v1:172.21.15.192:6789/0, reconnecting 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 >> v1:172.21.15.192:6789/0 conn(0x7fa8940a89e0 legacy=0x7fa89413ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.990+0000 7fa8927fc640 1 --2- 172.21.15.69:0/4063939925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa87c0039c0 0x7fa87c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 10 monclient(hunting): picked mon.a con 0x7fa87c0039c0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 1 --2- 172.21.15.69:0/4063939925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa87c0047c0 0x7fa87c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 10 monclient(hunting): picked mon.b con 0x7fa87c0047c0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 1 --2- 172.21.15.69:0/4063939925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa87c0052a0 0x7fa87c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 10 monclient(hunting): picked mon.c con 0x7fa87c0052a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940aaca0 con 0x7fa87c0039c0 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:25.998 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c00c1c0 con 0x7fa87c0047c0 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c00c380 con 0x7fa87c0052a0 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa8927fc640 10 monclient(hunting): _finish_auth 0 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.991+0000 7fa89a0b4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.992+0000 7fa893fff640 1 --2- 172.21.15.69:0/4063939925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa87c0052a0 0x7fa87c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.992+0000 7fa893fff640 10 monclient(hunting): get_auth_request con 0x7fa87c0052a0 auth_method 0 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.992+0000 7fa893fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.992+0000 7fa893fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.992+0000 7fa893fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.992+0000 7fa893fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.992+0000 7fa893fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:25.999 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.992+0000 7fa893fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-15T17:01:26.001 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 10 monclient(hunting): handle_auth_done global_id 4207 payload 293 2023-12-15T17:01:26.001 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 1 -- 172.21.15.69:0/4063939925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa87c0047c0 msgr2=0x7fa87c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:26.001 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 1 --2- 172.21.15.69:0/4063939925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa87c0047c0 0x7fa87c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:26.001 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 1 -- 172.21.15.69:0/4063939925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa87c0039c0 msgr2=0x7fa87c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:26.001 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 1 --2- 172.21.15.69:0/4063939925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa87c0039c0 0x7fa87c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:26.002 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 10 monclient: _finish_hunting 0 2023-12-15T17:01:26.002 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 1 monclient: found mon.c 2023-12-15T17:01:26.002 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:26.002 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:26.002 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 1 -- 172.21.15.69:0/4063939925 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c003520 con 0x7fa87c0052a0 2023-12-15T17:01:26.002 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 10 monclient: _finish_auth 0 2023-12-15T17:01:26.002 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 10 monclient: _check_auth_tickets 2023-12-15T17:01:26.002 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.994+0000 7fa893fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:26.002 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.995+0000 7fa893fff640 1 -- 172.21.15.69:0/4063939925 reap_dead start 2023-12-15T17:01:26.002 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:25.997+0000 7fa893fff640 1 --2- 172.21.15.69:0/4063939925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa87c0052a0 0x7fa87c00b980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa88c001e50 tx=0x7fa87c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fb465c05a384380 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:26.005 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.000+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c003b70 con 0x7fa87c0052a0 2023-12-15T17:01:26.005 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.000+0000 7fa8927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:26.005 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.000+0000 7fa8927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:26.005 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.000+0000 7fa8927fc640 10 monclient: dump: 2023-12-15T17:01:26.005 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:26.005 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:26.005 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.000+0000 7fa89a0b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.000+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88c003cf0 con 0x7fa87c0052a0 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.000+0000 7fa8927fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.000+0000 7fa89a0b4640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.000+0000 7fa89a0b4640 4 set_mon_vals no callback set 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.000+0000 7fa8927fc640 1 -- 172.21.15.69:0/4063939925 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c005310 con 0x7fa87c0052a0 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.000+0000 7fa8927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:26.006 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.001+0000 7fa8927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.001+0000 7fa8927fc640 10 monclient: dump: 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.001+0000 7fa89a0b4640 10 monclient: shutdown 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.001+0000 7fa89a0b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:26.007 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.001+0000 7fa89a0b4640 1 -- 172.21.15.69:0/4063939925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa87c0052a0 msgr2=0x7fa87c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.001+0000 7fa89a0b4640 1 --2- 172.21.15.69:0/4063939925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa87c0052a0 0x7fa87c00b980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa88c001e50 tx=0x7fa87c0057f0 comp rx=0 tx=0).stop 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.001+0000 7fa89a0b4640 1 -- 172.21.15.69:0/4063939925 shutdown_connections 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.001+0000 7fa89a0b4640 1 --2- 172.21.15.69:0/4063939925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa87c0052a0 0x7fa87c00b980 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.001+0000 7fa89a0b4640 1 -- 172.21.15.69:0/4063939925 >> 172.21.15.69:0/4063939925 conn(0x7fa8940b7940 msgr2=0x7fa8940b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.001+0000 7fa89a0b4640 1 -- 172.21.15.69:0/4063939925 shutdown_connections 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.001+0000 7fa89a0b4640 1 -- 172.21.15.69:0/4063939925 wait complete. 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.002+0000 7fa89a0b4640 10 monclient: build_initial_monmap 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.002+0000 7fa89a0b4640 10 monclient: monmap: 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:26.008 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:26.009 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:26.009 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:26.009 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:26.009 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:26.009 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.003+0000 7fa89a0b4640 1 librados: starting msgr at 2023-12-15T17:01:26.009 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.003+0000 7fa89a0b4640 1 librados: starting objecter 2023-12-15T17:01:26.009 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.003+0000 7fa89a0b4640 1 Processor -- start 2023-12-15T17:01:26.009 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.003+0000 7fa89a0b4640 1 -- start start 2023-12-15T17:01:26.009 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.003+0000 7fa89a0b4640 1 librados: setting wanted keys 2023-12-15T17:01:26.009 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.003+0000 7fa89a0b4640 1 librados: calling monclient init 2023-12-15T17:01:26.009 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.003+0000 7fa89a0b4640 10 monclient: init 2023-12-15T17:01:26.015 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:26.015 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:01:26.015 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa87c0052a0 0x7fa8940ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:26.015 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 10 monclient(hunting): picked mon.noname-b con 0x7fa87c0052a0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa894145ec0 0x7fa894143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 10 monclient(hunting): picked mon.noname-a con 0x7fa894145ec0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa894143a50 0x7fa8941da770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 10 monclient(hunting): picked mon.noname-c con 0x7fa894143a50 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940abca0 con 0x7fa894145ec0 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940ad4f0 con 0x7fa87c0052a0 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8941452f0 con 0x7fa894143a50 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.004+0000 7fa89a0b4640 10 monclient(hunting): authenticate will time out at 1390.321215s 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa8937fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa87c0052a0 0x7fa8940ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:26.016 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa892ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa894145ec0 0x7fa894143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa8937fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa87c0052a0 0x7fa8940ab6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:49794/0 (socket says 172.21.15.69:49794) 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa8937fe640 1 -- 172.21.15.69:0/2933551992 learned_addr learned my addr 172.21.15.69:0/2933551992 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa8937fe640 10 monclient(hunting): get_auth_request con 0x7fa87c0052a0 auth_method 0 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa8937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa8937fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa8937fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa893fff640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa894143a50 0x7fa8941da770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa893fff640 10 monclient(hunting): get_auth_request con 0x7fa894143a50 auth_method 0 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa893fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa893fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa893fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa892ffd640 10 monclient(hunting): get_auth_request con 0x7fa894145ec0 auth_method 0 2023-12-15T17:01:26.017 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa892ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa892ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.005+0000 7fa892ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.006+0000 7fa8937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.006+0000 7fa8937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.006+0000 7fa8937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.006+0000 7fa892ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.006+0000 7fa892ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.006+0000 7fa892ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.006+0000 7fa893fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.006+0000 7fa893fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.006+0000 7fa893fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.010+0000 7fa8937fe640 10 monclient(hunting): handle_auth_done global_id 4210 payload 824 2023-12-15T17:01:26.018 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.010+0000 7fa8937fe640 1 -- 172.21.15.69:0/2933551992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa894143a50 msgr2=0x7fa8941da770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:26.019 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.010+0000 7fa8937fe640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa894143a50 0x7fa8941da770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:26.019 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.010+0000 7fa8937fe640 1 -- 172.21.15.69:0/2933551992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa894145ec0 msgr2=0x7fa894143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:26.019 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.010+0000 7fa8937fe640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa894145ec0 0x7fa894143510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:26.019 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.010+0000 7fa8937fe640 10 monclient: _finish_hunting 0 2023-12-15T17:01:26.019 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.010+0000 7fa8937fe640 1 monclient: found mon.noname-b 2023-12-15T17:01:26.019 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.010+0000 7fa8937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:26.019 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.010+0000 7fa8937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:01:26.019 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.010+0000 7fa8937fe640 1 -- 172.21.15.69:0/2933551992 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8941462e0 con 0x7fa87c0052a0 2023-12-15T17:01:26.019 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.013+0000 7fa8937fe640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa87c0052a0 0x7fa8940ab6a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa88400c730 tx=0x7fa88400cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:26.019 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.013+0000 7fa87bfff640 10 client.?.objecter ms_handle_connect 0x7fa87c0052a0 2023-12-15T17:01:26.019 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.013+0000 7fa87bfff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:26.020 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:17.449 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:26.022 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 1 -- 172.21.15.69:0/2933551992 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8840074e0 con 0x7fa87c0052a0 2023-12-15T17:01:26.022 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:26.022 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:26.022 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 10 monclient: dump: 2023-12-15T17:01:26.022 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:26.022 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 10 monclient: _finish_auth 0 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 10 monclient: _check_auth_tickets 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 1 -- 172.21.15.69:0/2933551992 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa884007640 con 0x7fa87c0052a0 2023-12-15T17:01:26.023 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa89a0b4640 5 monclient: authenticate success, global_id 4210 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa89a0b4640 10 monclient: _renew_subs 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa89a0b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa89a0b4640 1 -- 172.21.15.69:0/2933551992 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa894143e90 con 0x7fa87c0052a0 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 1 -- 172.21.15.69:0/2933551992 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884004a00 con 0x7fa87c0052a0 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa87bfff640 10 monclient: dump: 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:26.024 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa891ffb640 4 set_mon_vals no callback set 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa89a0b4640 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa89a0b4640 10 monclient: _renew_subs 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa89a0b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa89a0b4640 1 -- 172.21.15.69:0/2933551992 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8941dacb0 con 0x7fa87c0052a0 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.019+0000 7fa89a0b4640 1 librados: init done 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.020+0000 7fa879ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.020+0000 7fa879ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.020+0000 7fa879ffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:26.025 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.020+0000 7fa879ffb640 1 -- 172.21.15.69:0/2933551992 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c005040 con 0x7fa87c0052a0 2023-12-15T17:01:26.057 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.054+0000 7fa87bfff640 1 -- 172.21.15.69:0/2933551992 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 4) v1 ==== 89314+0+0 (secure 0 0 0) 0x7fa88400a5e0 con 0x7fa87c0052a0 2023-12-15T17:01:26.057 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.054+0000 7fa87bfff640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa88006cac0 0x7fa88006ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:26.057 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.054+0000 7fa892ffd640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa88006cac0 0x7fa88006ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:26.057 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.055+0000 7fa892ffd640 10 monclient: get_auth_request con 0x7fa88006cac0 auth_method 0 2023-12-15T17:01:26.057 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.055+0000 7fa892ffd640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa88006cac0 0x7fa88006ef80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa888006fd0 tx=0x7fa888008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:26.057 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.055+0000 7fa87bfff640 10 client.4210.objecter ms_handle_connect 0x7fa88006cac0 2023-12-15T17:01:26.206 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:17.636 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:26.300 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:17.723 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:26.351 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.349+0000 7fa87bfff640 1 -- 172.21.15.69:0/2933551992 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa88400ce40 con 0x7fa87c0052a0 2023-12-15T17:01:26.351 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.349+0000 7fa87bfff640 10 client.4210.objecter ms_dispatch 0x7fa894000b90 osd_map(10..10 src has 1..10) v4 2023-12-15T17:01:26.351 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.349+0000 7fa87bfff640 3 client.4210.objecter handle_osd_map got epochs [10,10] > 0 2023-12-15T17:01:26.351 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.349+0000 7fa87bfff640 3 client.4210.objecter handle_osd_map decoding full epoch 10 2023-12-15T17:01:26.351 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.349+0000 7fa87bfff640 20 client.4210.objecter dump_active .. 0 homeless 2023-12-15T17:01:26.369 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.367+0000 7fa87bfff640 1 -- 172.21.15.69:0/2933551992 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa884054220 con 0x7fa87c0052a0 2023-12-15T17:01:26.369 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.367+0000 7fa87bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:26.369 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.367+0000 7fa87bfff640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:26.472 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:17.910 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:26.522 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:17.951 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:26.549 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:17.980 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:26.625 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.623+0000 7fa879ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:26.625 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.623+0000 7fa879ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:26.625 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.623+0000 7fa879ffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:26.625 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.623+0000 7fa879ffb640 1 -- 172.21.15.69:0/2933551992 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa86c0055c0 con 0x7fa87c0052a0 2023-12-15T17:01:26.654 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.652+0000 7fa87bfff640 1 -- 172.21.15.69:0/2933551992 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fa884018030 con 0x7fa87c0052a0 2023-12-15T17:01:26.654 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.652+0000 7fa87bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:26.654 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.652+0000 7fa87bfff640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:26.654 INFO:teuthology.misc.health.smithi069.stdout: 2023-12-15T17:01:26.654 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"1a9f4b8c-183d-4435-93ef-5057f2b4aded","created":"2023-12-15T17:01:02.006221+0000","modified":"2023-12-15T17:01:08.340674+0000","last_up_change":"0.000000","last_in_change":"2023-12-15T17:01:08.340674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51cce62a-d177-4e4b-a814-7e537407f397","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1556db3a-b386-444f-8327-3dce2ad46d33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eaac40b5-9fc0-4e13-a322-763a9f3b685f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"71221e7e-8713-4fc4-bef1-c1cbe40627da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2c9feecc-d3b5-429d-98fe-e6eddc91eb8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac6e8a35-6df7-48d4-b59a-1f87eceaacfa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T17:01:26.657 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.654+0000 7fa879ffb640 10 librados: watch_flush enter 2023-12-15T17:01:26.657 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 10 librados: watch_flush exit 2023-12-15T17:01:26.657 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 20 client.4210.objecter shutdown clearing up homeless session... 2023-12-15T17:01:26.657 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 10 client.4210.objecter successfully canceled tick 2023-12-15T17:01:26.657 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 -- 172.21.15.69:0/2933551992 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa88006cac0 msgr2=0x7fa88006ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:26.657 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa88006cac0 0x7fa88006ef80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa888006fd0 tx=0x7fa888008040 comp rx=0 tx=0).stop 2023-12-15T17:01:26.657 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 10 monclient: shutdown 2023-12-15T17:01:26.657 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:26.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 -- 172.21.15.69:0/2933551992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa87c0052a0 msgr2=0x7fa8940ab6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:26.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa87c0052a0 0x7fa8940ab6a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa88400c730 tx=0x7fa88400cbf0 comp rx=0 tx=0).stop 2023-12-15T17:01:26.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 -- 172.21.15.69:0/2933551992 shutdown_connections 2023-12-15T17:01:26.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa88006cac0 0x7fa88006ef80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:26.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa894143a50 0x7fa8941da770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:26.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa894145ec0 0x7fa894143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:26.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 --2- 172.21.15.69:0/2933551992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa87c0052a0 0x7fa8940ab6a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:26.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 -- 172.21.15.69:0/2933551992 >> 172.21.15.69:0/2933551992 conn(0x7fa8940b7940 msgr2=0x7fa894073c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:26.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 -- 172.21.15.69:0/2933551992 shutdown_connections 2023-12-15T17:01:26.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 -- 172.21.15.69:0/2933551992 wait complete. 2023-12-15T17:01:26.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:26.655+0000 7fa879ffb640 1 librados: shutdown 2023-12-15T17:01:26.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-15T17:01:26.933 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:18.363 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:31.778 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:23.208 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:31.782 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:23.212 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.042 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:23.454 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.049 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:23.460 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.072 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:23.495 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.076 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:23.499 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.136 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:23.566 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.139 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:23.570 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.346 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:23.783 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.349 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:23.781 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.350 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:23.787 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.353 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:23.784 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.374 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:23.805 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.380 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:23.811 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.552 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:23.981 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:32.556 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:23.985 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:33.476 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-15T17:01:33.565 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.563+0000 7fc934133640 10 monclient: get_monmap_and_config 2023-12-15T17:01:33.565 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.563+0000 7fc934133640 10 monclient: build_initial_monmap 2023-12-15T17:01:33.565 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.563+0000 7fc934133640 10 monclient: monmap: 2023-12-15T17:01:33.565 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:33.565 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:33.566 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:01:33.565109+0000 2023-12-15T17:01:33.566 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:01:33.565109+0000 2023-12-15T17:01:33.566 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:33.566 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:33.566 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:33.566 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:33.566 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:33.566 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:33.566 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:33.566 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:33.568 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.564+0000 7fc934133640 1 Processor -- start 2023-12-15T17:01:33.568 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.564+0000 7fc934133640 1 -- start start 2023-12-15T17:01:33.568 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.564+0000 7fc934133640 10 monclient: init 2023-12-15T17:01:33.568 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.564+0000 7fc934133640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:33.568 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-12-15T17:01:33.568 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc92c0a89e0 0x7fc92c13ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:33.568 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 10 monclient(hunting): picked mon.noname-c con 0x7fc92c0a89e0 addr v2:172.21.15.192:3300/0 2023-12-15T17:01:33.568 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c13f460 0x7fc92c1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 10 monclient(hunting): picked mon.noname-b con 0x7fc92c13f460 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc92c142710 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92c0aa770 con 0x7fc92c142710 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc92c13f860 con 0x7fc92c0a89e0 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc92c0aa9c0 con 0x7fc92c13f460 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc934133640 10 monclient(hunting): authenticate will time out at 1397.881484s 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.565+0000 7fc9316ce640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c13f460 0x7fc92c1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc931ecf640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc92c0a89e0 0x7fc92c13ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc9326d0640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fc92c142710 0x7fc92c145470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:47418/0 (socket says 172.21.15.69:47418) 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc9326d0640 1 -- 172.21.15.69:0/1275986179 learned_addr learned my addr 172.21.15.69:0/1275986179 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:01:33.569 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc9316ce640 10 monclient(hunting): get_auth_request con 0x7fc92c13f460 auth_method 0 2023-12-15T17:01:33.570 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc9316ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:33.570 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc9316ce640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:33.570 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc9316ce640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:33.570 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc931ecf640 10 monclient(hunting): get_auth_request con 0x7fc92c0a89e0 auth_method 0 2023-12-15T17:01:33.570 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc931ecf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:33.570 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc931ecf640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:33.570 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc931ecf640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:33.570 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc9316ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:33.570 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc9316ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:33.570 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.566+0000 7fc9316ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.567+0000 7fc931ecf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.567+0000 7fc931ecf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.567+0000 7fc931ecf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.568+0000 7fc9316ce640 10 monclient(hunting): handle_auth_done global_id 4190 payload 274 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.568+0000 7fc9316ce640 1 -- 172.21.15.69:0/1275986179 >> v2:172.21.15.192:3300/0 conn(0x7fc92c0a89e0 msgr2=0x7fc92c13ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.568+0000 7fc9316ce640 1 --2- 172.21.15.69:0/1275986179 >> v2:172.21.15.192:3300/0 conn(0x7fc92c0a89e0 0x7fc92c13ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.568+0000 7fc9316ce640 1 -- 172.21.15.69:0/1275986179 >> v1:172.21.15.192:6789/0 conn(0x7fc92c142710 legacy=0x7fc92c145470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.568+0000 7fc9316ce640 10 monclient: _finish_hunting 0 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.568+0000 7fc9316ce640 1 monclient: found mon.noname-b 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.568+0000 7fc9316ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.568+0000 7fc9316ce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:01:33.571 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.568+0000 7fc9316ce640 1 -- 172.21.15.69:0/1275986179 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92c0aaca0 con 0x7fc92c13f460 2023-12-15T17:01:33.572 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.570+0000 7fc9316ce640 1 --2- 172.21.15.69:0/1275986179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c13f460 0x7fc92c1421d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0097a0 tx=0x7fc91c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad1f89572e67f6eb server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:33.574 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.572+0000 7fc930ecd640 1 -- 172.21.15.69:0/1275986179 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c02e070 con 0x7fc92c13f460 2023-12-15T17:01:33.574 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.572+0000 7fc930ecd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:33.574 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.572+0000 7fc930ecd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:33.574 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.572+0000 7fc930ecd640 10 monclient: dump: 2023-12-15T17:01:33.574 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:33.574 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:33.574 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:33.574 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:33.575 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:33.575 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:33.575 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:33.575 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:33.575 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:33.575 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:33.575 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.572+0000 7fc930ecd640 10 monclient: _finish_auth 0 2023-12-15T17:01:33.575 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.572+0000 7fc930ecd640 10 monclient: _check_auth_tickets 2023-12-15T17:01:33.575 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.572+0000 7fc930ecd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:33.575 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.572+0000 7fc934133640 5 monclient: authenticate success, global_id 4190 2023-12-15T17:01:33.575 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.572+0000 7fc934133640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.574+0000 7fc930ecd640 1 -- 172.21.15.69:0/1275986179 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91c0315a0 con 0x7fc92c13f460 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.574+0000 7fc930ecd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.574+0000 7fc930ecd640 1 -- 172.21.15.69:0/1275986179 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c031970 con 0x7fc92c13f460 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.574+0000 7fc934133640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.574+0000 7fc934133640 4 set_mon_vals no callback set 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.574+0000 7fc930ecd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.574+0000 7fc930ecd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.574+0000 7fc930ecd640 10 monclient: dump: 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:33.576 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:33.577 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:33.577 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:33.577 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:33.577 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:33.577 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:33.577 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:33.578 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.575+0000 7fc934133640 10 monclient: shutdown 2023-12-15T17:01:33.578 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.575+0000 7fc934133640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:33.578 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.575+0000 7fc934133640 1 -- 172.21.15.69:0/1275986179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c13f460 msgr2=0x7fc92c1421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:33.578 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.575+0000 7fc934133640 1 --2- 172.21.15.69:0/1275986179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c13f460 0x7fc92c1421d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0097a0 tx=0x7fc91c02b840 comp rx=0 tx=0).stop 2023-12-15T17:01:33.578 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.575+0000 7fc934133640 1 -- 172.21.15.69:0/1275986179 shutdown_connections 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.575+0000 7fc934133640 1 --2- 172.21.15.69:0/1275986179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c13f460 0x7fc92c1421d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.575+0000 7fc934133640 1 --2- 172.21.15.69:0/1275986179 >> v2:172.21.15.192:3300/0 conn(0x7fc92c0a89e0 0x7fc92c13ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.575+0000 7fc934133640 1 -- 172.21.15.69:0/1275986179 >> 172.21.15.69:0/1275986179 conn(0x7fc92c0b7940 msgr2=0x7fc92c0b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.575+0000 7fc934133640 1 -- 172.21.15.69:0/1275986179 shutdown_connections 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.575+0000 7fc934133640 1 -- 172.21.15.69:0/1275986179 wait complete. 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.576+0000 7fc934133640 10 monclient: build_initial_monmap 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.576+0000 7fc934133640 10 monclient: monmap: 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:33.579 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:33.580 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:33.580 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:33.580 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.576+0000 7fc934133640 1 librados: starting msgr at 2023-12-15T17:01:33.580 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.577+0000 7fc934133640 1 librados: starting objecter 2023-12-15T17:01:33.580 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.577+0000 7fc934133640 1 Processor -- start 2023-12-15T17:01:33.580 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.577+0000 7fc934133640 1 -- start start 2023-12-15T17:01:33.580 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.577+0000 7fc934133640 1 librados: setting wanted keys 2023-12-15T17:01:33.580 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.577+0000 7fc934133640 1 librados: calling monclient init 2023-12-15T17:01:33.580 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.577+0000 7fc934133640 10 monclient: init 2023-12-15T17:01:33.583 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.577+0000 7fc934133640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:33.583 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:01:33.583 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c0a89e0 0x7fc92c145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:33.583 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 10 monclient(hunting): picked mon.noname-c con 0x7fc92c0a89e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:33.583 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc92c13f460 0x7fc92c1d6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:33.583 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 10 monclient(hunting): picked mon.noname-b con 0x7fc92c13f460 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:33.583 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c142710 0x7fc92c1d70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:33.583 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 10 monclient(hunting): picked mon.noname-a con 0x7fc92c142710 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92c1d47e0 con 0x7fc92c142710 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92c145db0 con 0x7fc92c13f460 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc92c0ad8d0 con 0x7fc92c0a89e0 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc934133640 10 monclient(hunting): authenticate will time out at 1397.894599s 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc9316ce640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc92c13f460 0x7fc92c1d6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc931ecf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c0a89e0 0x7fc92c145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc9316ce640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc92c13f460 0x7fc92c1d6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:42508/0 (socket says 172.21.15.69:42508) 2023-12-15T17:01:33.584 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc9316ce640 1 -- 172.21.15.69:0/3506601086 learned_addr learned my addr 172.21.15.69:0/3506601086 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:33.585 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc9316ce640 10 monclient(hunting): get_auth_request con 0x7fc92c13f460 auth_method 0 2023-12-15T17:01:33.585 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc9316ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:33.585 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc9316ce640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:33.585 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc9316ce640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:33.585 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.578+0000 7fc9326d0640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c142710 0x7fc92c1d70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:33.585 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc931ecf640 10 monclient(hunting): get_auth_request con 0x7fc92c0a89e0 auth_method 0 2023-12-15T17:01:33.586 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc931ecf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:33.586 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc931ecf640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:33.586 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc931ecf640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:33.586 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc9316ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:33.586 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc9316ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:33.586 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc9316ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:33.586 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc9326d0640 10 monclient(hunting): get_auth_request con 0x7fc92c142710 auth_method 0 2023-12-15T17:01:33.587 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc9326d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:33.587 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc9326d0640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:33.587 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc9326d0640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:33.587 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc931ecf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:33.587 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc931ecf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:33.587 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc931ecf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:33.587 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc9326d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:33.588 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc9326d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:33.588 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.579+0000 7fc9326d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:33.589 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.583+0000 7fc931ecf640 10 monclient(hunting): handle_auth_done global_id 4193 payload 824 2023-12-15T17:01:33.589 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.583+0000 7fc931ecf640 1 -- 172.21.15.69:0/3506601086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc92c13f460 msgr2=0x7fc92c1d6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:33.589 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.583+0000 7fc931ecf640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc92c13f460 0x7fc92c1d6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:33.589 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.583+0000 7fc931ecf640 1 -- 172.21.15.69:0/3506601086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c142710 msgr2=0x7fc92c1d70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:33.589 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.583+0000 7fc931ecf640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c142710 0x7fc92c1d70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:33.589 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.583+0000 7fc931ecf640 10 monclient: _finish_hunting 0 2023-12-15T17:01:33.589 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.583+0000 7fc931ecf640 1 monclient: found mon.noname-c 2023-12-15T17:01:33.590 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.583+0000 7fc931ecf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:33.590 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.583+0000 7fc931ecf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:01:33.590 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.583+0000 7fc931ecf640 1 -- 172.21.15.69:0/3506601086 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91c009400 con 0x7fc92c0a89e0 2023-12-15T17:01:33.590 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.586+0000 7fc931ecf640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c0a89e0 0x7fc92c145280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc92800e870 tx=0x7fc92800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:33.590 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.586+0000 7fc9227fc640 10 client.?.objecter ms_handle_connect 0x7fc92c0a89e0 2023-12-15T17:01:33.590 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.586+0000 7fc9227fc640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:33.594 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 1 -- 172.21.15.69:0/3506601086 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92800ee60 con 0x7fc92c0a89e0 2023-12-15T17:01:33.594 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:33.594 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:01:33.594 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 10 monclient: dump: 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 10 monclient: _finish_auth 0 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 10 monclient: _check_auth_tickets 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:33.595 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 1 -- 172.21.15.69:0/3506601086 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92800c940 con 0x7fc92c0a89e0 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 1 -- 172.21.15.69:0/3506601086 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92800ccd0 con 0x7fc92c0a89e0 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc9227fc640 10 monclient: dump: 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:33.596 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:33.597 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:33.597 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.590+0000 7fc923fff640 4 set_mon_vals no callback set 2023-12-15T17:01:33.597 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.591+0000 7fc934133640 5 monclient: authenticate success, global_id 4193 2023-12-15T17:01:33.597 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.591+0000 7fc934133640 10 monclient: _renew_subs 2023-12-15T17:01:33.597 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.591+0000 7fc934133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:33.597 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.591+0000 7fc934133640 1 -- 172.21.15.69:0/3506601086 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92c0ade30 con 0x7fc92c0a89e0 2023-12-15T17:01:33.597 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.591+0000 7fc934133640 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:33.598 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.591+0000 7fc934133640 10 monclient: _renew_subs 2023-12-15T17:01:33.598 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.591+0000 7fc934133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:33.598 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.591+0000 7fc934133640 1 -- 172.21.15.69:0/3506601086 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc92c0ad5d0 con 0x7fc92c0a89e0 2023-12-15T17:01:33.598 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.591+0000 7fc934133640 1 librados: init done 2023-12-15T17:01:33.598 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.592+0000 7fc934133640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:33.598 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.592+0000 7fc934133640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:33.598 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.592+0000 7fc934133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:33.598 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.592+0000 7fc934133640 1 -- 172.21.15.69:0/3506601086 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc92c069c00 con 0x7fc92c0a89e0 2023-12-15T17:01:33.615 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.612+0000 7fc9227fc640 1 -- 172.21.15.69:0/3506601086 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 4) v1 ==== 89314+0+0 (secure 0 0 0) 0x7fc928013470 con 0x7fc92c0a89e0 2023-12-15T17:01:33.615 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.613+0000 7fc9227fc640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc91006cb40 0x7fc91006f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:33.615 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.613+0000 7fc9316ce640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc91006cb40 0x7fc91006f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:33.615 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.613+0000 7fc9316ce640 10 monclient: get_auth_request con 0x7fc91006cb40 auth_method 0 2023-12-15T17:01:33.616 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.614+0000 7fc9227fc640 10 client.4193.objecter ms_handle_connect 0x7fc91006cb40 2023-12-15T17:01:33.616 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.614+0000 7fc9316ce640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc91006cb40 0x7fc91006f000 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0097a0 tx=0x7fc91c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:33.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.656+0000 7fc9227fc640 1 -- 172.21.15.69:0/3506601086 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc92800ce30 con 0x7fc92c0a89e0 2023-12-15T17:01:33.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.656+0000 7fc9227fc640 10 client.4193.objecter ms_dispatch 0x7fc92c000b90 osd_map(10..10 src has 1..10) v4 2023-12-15T17:01:33.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.656+0000 7fc9227fc640 3 client.4193.objecter handle_osd_map got epochs [10,10] > 0 2023-12-15T17:01:33.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.656+0000 7fc9227fc640 3 client.4193.objecter handle_osd_map decoding full epoch 10 2023-12-15T17:01:33.658 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.656+0000 7fc9227fc640 20 client.4193.objecter dump_active .. 0 homeless 2023-12-15T17:01:33.860 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.858+0000 7fc9227fc640 1 -- 172.21.15.69:0/3506601086 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc928054eb0 con 0x7fc92c0a89e0 2023-12-15T17:01:33.860 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.859+0000 7fc9227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:33.861 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:33.859+0000 7fc9227fc640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:34.154 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.152+0000 7fc934133640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:34.154 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.152+0000 7fc934133640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:34.155 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.152+0000 7fc934133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:34.155 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.152+0000 7fc934133640 1 -- 172.21.15.69:0/3506601086 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc92c0ab010 con 0x7fc92c0a89e0 2023-12-15T17:01:34.181 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.179+0000 7fc9227fc640 1 -- 172.21.15.69:0/3506601086 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fc928021090 con 0x7fc92c0a89e0 2023-12-15T17:01:34.181 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.179+0000 7fc9227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:34.182 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.179+0000 7fc9227fc640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:34.182 INFO:teuthology.misc.health.smithi069.stdout: 2023-12-15T17:01:34.182 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"1a9f4b8c-183d-4435-93ef-5057f2b4aded","created":"2023-12-15T17:01:02.006221+0000","modified":"2023-12-15T17:01:08.340674+0000","last_up_change":"0.000000","last_in_change":"2023-12-15T17:01:08.340674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51cce62a-d177-4e4b-a814-7e537407f397","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1556db3a-b386-444f-8327-3dce2ad46d33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eaac40b5-9fc0-4e13-a322-763a9f3b685f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"71221e7e-8713-4fc4-bef1-c1cbe40627da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2c9feecc-d3b5-429d-98fe-e6eddc91eb8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac6e8a35-6df7-48d4-b59a-1f87eceaacfa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.181+0000 7fc934133640 10 librados: watch_flush enter 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 10 librados: watch_flush exit 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 20 client.4193.objecter shutdown clearing up homeless session... 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 10 client.4193.objecter successfully canceled tick 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 -- 172.21.15.69:0/3506601086 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc91006cb40 msgr2=0x7fc91006f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc91006cb40 0x7fc91006f000 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0097a0 tx=0x7fc91c03d040 comp rx=0 tx=0).stop 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 10 monclient: shutdown 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 -- 172.21.15.69:0/3506601086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c0a89e0 msgr2=0x7fc92c145280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c0a89e0 0x7fc92c145280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc92800e870 tx=0x7fc92800ed30 comp rx=0 tx=0).stop 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 -- 172.21.15.69:0/3506601086 shutdown_connections 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc91006cb40 0x7fc91006f000 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:34.185 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c142710 0x7fc92c1d70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:34.186 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc92c13f460 0x7fc92c1d6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:34.186 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 --2- 172.21.15.69:0/3506601086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc92c0a89e0 0x7fc92c145280 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:34.186 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 -- 172.21.15.69:0/3506601086 >> 172.21.15.69:0/3506601086 conn(0x7fc92c0b7940 msgr2=0x7fc92c0b7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:34.186 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 -- 172.21.15.69:0/3506601086 shutdown_connections 2023-12-15T17:01:34.186 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 -- 172.21.15.69:0/3506601086 wait complete. 2023-12-15T17:01:34.186 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:34.182+0000 7fc934133640 1 librados: shutdown 2023-12-15T17:01:34.195 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-15T17:01:35.806 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:27.236 86807== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:36.331 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:27.768 86753== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:36.357 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:27.788 86806== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:36.371 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:27.802 86754== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:36.410 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:27.822 86809== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:36.567 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:27.990 86756== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:36.665 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:28.094 86808== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:36.733 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:28.164 86755== Warning: unimplemented fcntl command: 1038 2023-12-15T17:01:41.001 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-15T17:01:41.099 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.098+0000 7fd749ce7640 10 monclient: get_monmap_and_config 2023-12-15T17:01:41.099 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.098+0000 7fd749ce7640 10 monclient: build_initial_monmap 2023-12-15T17:01:41.099 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.098+0000 7fd749ce7640 10 monclient: monmap: 2023-12-15T17:01:41.099 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:41.099 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:41.099 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:01:41.098800+0000 2023-12-15T17:01:41.100 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:01:41.098800+0000 2023-12-15T17:01:41.100 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:41.100 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:41.100 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:41.100 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:41.100 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:41.100 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:41.100 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:41.100 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:41.102 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 1 Processor -- start 2023-12-15T17:01:41.102 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 1 -- start start 2023-12-15T17:01:41.102 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 10 monclient: init 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7440a89e0 0x7fd74413ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 10 monclient(hunting): picked mon.noname-b con 0x7fd7440a89e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd74413f460 0x7fd7441421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 10 monclient(hunting): picked mon.noname-a con 0x7fd74413f460 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd744142710 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7440aa770 con 0x7fd744142710 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.099+0000 7fd749ce7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd74413f860 con 0x7fd74413f460 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.100+0000 7fd749ce7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.100+0000 7fd749ce7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7440aa9c0 con 0x7fd7440a89e0 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.100+0000 7fd749ce7640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:41.103 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.100+0000 7fd749ce7640 10 monclient(hunting): authenticate will time out at 1405.415729s 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.100+0000 7fd742ffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd74413f460 0x7fd7441421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.100+0000 7fd743fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fd744142710 0x7fd744145470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:58994/0 (socket says 172.21.15.69:58994) 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.100+0000 7fd743fff640 1 -- 172.21.15.69:0/3887367160 learned_addr learned my addr 172.21.15.69:0/3887367160 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd7437fe640 1 --2- 172.21.15.69:0/3887367160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7440a89e0 0x7fd74413ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd742ffd640 10 monclient(hunting): get_auth_request con 0x7fd74413f460 auth_method 0 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd742ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd742ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd7437fe640 10 monclient(hunting): get_auth_request con 0x7fd7440a89e0 auth_method 0 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd7437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd7437fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:41.104 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd7437fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:41.105 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd742ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:41.105 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd742ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:41.105 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd742ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:41.105 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd7437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:41.105 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd7437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:41.105 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.101+0000 7fd7437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:41.105 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.103+0000 7fd742ffd640 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2023-12-15T17:01:41.105 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.103+0000 7fd742ffd640 1 -- 172.21.15.69:0/3887367160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7440a89e0 msgr2=0x7fd74413ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:41.105 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.103+0000 7fd742ffd640 1 --2- 172.21.15.69:0/3887367160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7440a89e0 0x7fd74413ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:41.106 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.103+0000 7fd742ffd640 1 -- 172.21.15.69:0/3887367160 >> v1:172.21.15.192:6789/0 conn(0x7fd744142710 legacy=0x7fd744145470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:41.106 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.103+0000 7fd742ffd640 10 monclient: _finish_hunting 0 2023-12-15T17:01:41.106 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.103+0000 7fd742ffd640 1 monclient: found mon.noname-a 2023-12-15T17:01:41.106 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.103+0000 7fd742ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:41.106 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.103+0000 7fd742ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:01:41.106 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.103+0000 7fd742ffd640 1 -- 172.21.15.69:0/3887367160 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7440aaca0 con 0x7fd74413f460 2023-12-15T17:01:41.106 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.105+0000 7fd742ffd640 1 --2- 172.21.15.69:0/3887367160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd74413f460 0x7fd7441421d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd7380097a0 tx=0x7fd73802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6196e1edeb3b837f server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:41.110 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 1 -- 172.21.15.69:0/3887367160 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73802e070 con 0x7fd74413f460 2023-12-15T17:01:41.110 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:41.110 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:01:41.110 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 10 monclient: dump: 2023-12-15T17:01:41.110 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:41.110 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:41.110 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:41.110 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 10 monclient: _finish_auth 0 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 10 monclient: _check_auth_tickets 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 1 -- 172.21.15.69:0/3887367160 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7380315a0 con 0x7fd74413f460 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 1 -- 172.21.15.69:0/3887367160 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738031970 con 0x7fd74413f460 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd749ce7640 5 monclient: authenticate success, global_id 4209 2023-12-15T17:01:41.111 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd749ce7640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd749ce7640 4 set_mon_vals no callback set 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.109+0000 7fd7427fc640 10 monclient: dump: 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:41.112 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:41.115 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.110+0000 7fd749ce7640 10 monclient: shutdown 2023-12-15T17:01:41.115 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.110+0000 7fd749ce7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:41.115 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.110+0000 7fd749ce7640 1 -- 172.21.15.69:0/3887367160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd74413f460 msgr2=0x7fd7441421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:41.115 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.110+0000 7fd749ce7640 1 --2- 172.21.15.69:0/3887367160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd74413f460 0x7fd7441421d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd7380097a0 tx=0x7fd73802b840 comp rx=0 tx=0).stop 2023-12-15T17:01:41.115 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.110+0000 7fd749ce7640 1 -- 172.21.15.69:0/3887367160 shutdown_connections 2023-12-15T17:01:41.115 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.110+0000 7fd749ce7640 1 --2- 172.21.15.69:0/3887367160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd74413f460 0x7fd7441421d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:41.115 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.110+0000 7fd749ce7640 1 --2- 172.21.15.69:0/3887367160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7440a89e0 0x7fd74413ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:41.115 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.110+0000 7fd749ce7640 1 -- 172.21.15.69:0/3887367160 >> 172.21.15.69:0/3887367160 conn(0x7fd7440b7940 msgr2=0x7fd7440b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:41.115 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.110+0000 7fd749ce7640 1 -- 172.21.15.69:0/3887367160 shutdown_connections 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.110+0000 7fd749ce7640 1 -- 172.21.15.69:0/3887367160 wait complete. 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.111+0000 7fd749ce7640 10 monclient: build_initial_monmap 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.111+0000 7fd749ce7640 10 monclient: monmap: 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.112+0000 7fd749ce7640 1 librados: starting msgr at 2023-12-15T17:01:41.116 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.112+0000 7fd749ce7640 1 librados: starting objecter 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.112+0000 7fd749ce7640 1 Processor -- start 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.112+0000 7fd749ce7640 1 -- start start 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.112+0000 7fd749ce7640 1 librados: setting wanted keys 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.112+0000 7fd749ce7640 1 librados: calling monclient init 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.112+0000 7fd749ce7640 10 monclient: init 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7440a89e0 0x7fd7440abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 10 monclient(hunting): picked mon.noname-b con 0x7fd7440a89e0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd74413f460 0x7fd7440ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 10 monclient(hunting): picked mon.noname-c con 0x7fd74413f460 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd744142710 0x7fd7441dabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 10 monclient(hunting): picked mon.noname-a con 0x7fd744142710 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:41.117 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd744145ef0 con 0x7fd744142710 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7441da6d0 con 0x7fd7440a89e0 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7440ac0f0 con 0x7fd74413f460 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.113+0000 7fd749ce7640 10 monclient(hunting): authenticate will time out at 1405.429258s 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd7437fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7440a89e0 0x7fd7440abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd742ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd74413f460 0x7fd7440ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd743fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd744142710 0x7fd7441dabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd742ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd74413f460 0x7fd7440ac930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37072/0 (socket says 172.21.15.69:37072) 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd742ffd640 1 -- 172.21.15.69:0/858499531 learned_addr learned my addr 172.21.15.69:0/858499531 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd742ffd640 10 monclient(hunting): get_auth_request con 0x7fd74413f460 auth_method 0 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:41.118 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd742ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:41.119 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd742ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:41.119 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd7437fe640 10 monclient(hunting): get_auth_request con 0x7fd7440a89e0 auth_method 0 2023-12-15T17:01:41.119 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd7437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:41.119 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd7437fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:41.119 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd7437fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:41.119 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd743fff640 10 monclient(hunting): get_auth_request con 0x7fd744142710 auth_method 0 2023-12-15T17:01:41.119 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd743fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:41.119 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd743fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:41.119 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.114+0000 7fd743fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:41.121 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.115+0000 7fd742ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:41.121 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.115+0000 7fd742ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:41.121 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.115+0000 7fd742ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:41.121 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.115+0000 7fd7437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.115+0000 7fd7437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.115+0000 7fd7437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.115+0000 7fd743fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.115+0000 7fd743fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.115+0000 7fd743fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.117+0000 7fd742ffd640 10 monclient(hunting): handle_auth_done global_id 4199 payload 824 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.117+0000 7fd742ffd640 1 -- 172.21.15.69:0/858499531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7440a89e0 msgr2=0x7fd7440abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.117+0000 7fd742ffd640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7440a89e0 0x7fd7440abbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.117+0000 7fd742ffd640 1 -- 172.21.15.69:0/858499531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd744142710 msgr2=0x7fd7441dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.117+0000 7fd742ffd640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd744142710 0x7fd7441dabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.117+0000 7fd742ffd640 10 monclient: _finish_hunting 0 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.117+0000 7fd742ffd640 1 monclient: found mon.noname-c 2023-12-15T17:01:41.122 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.117+0000 7fd742ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:41.123 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.117+0000 7fd742ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:01:41.123 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.117+0000 7fd742ffd640 1 -- 172.21.15.69:0/858499531 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd738009400 con 0x7fd74413f460 2023-12-15T17:01:41.123 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.120+0000 7fd742ffd640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd74413f460 0x7fd7440ac930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd7380096f0 tx=0x7fd7380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:41.123 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.120+0000 7fd72bfff640 10 client.?.objecter ms_handle_connect 0x7fd74413f460 2023-12-15T17:01:41.123 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.120+0000 7fd72bfff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:41.128 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.124+0000 7fd72bfff640 1 -- 172.21.15.69:0/858499531 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738032840 con 0x7fd74413f460 2023-12-15T17:01:41.128 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.124+0000 7fd72bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:41.128 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.124+0000 7fd72bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:01:41.128 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.124+0000 7fd72bfff640 10 monclient: dump: 2023-12-15T17:01:41.128 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:41.128 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:41.128 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:41.128 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:41.128 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:41.128 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:41.128 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd72bfff640 10 monclient: _finish_auth 0 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd72bfff640 10 monclient: _check_auth_tickets 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd72bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd72bfff640 1 -- 172.21.15.69:0/858499531 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7380329c0 con 0x7fd74413f460 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd72bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd72bfff640 1 -- 172.21.15.69:0/858499531 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738032d70 con 0x7fd74413f460 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd72bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd72bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd72bfff640 10 monclient: dump: 2023-12-15T17:01:41.129 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd749ce7640 5 monclient: authenticate success, global_id 4199 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd749ce7640 10 monclient: _renew_subs 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd749ce7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd749ce7640 1 -- 172.21.15.69:0/858499531 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7440ace70 con 0x7fd74413f460 2023-12-15T17:01:41.130 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd741ffb640 4 set_mon_vals no callback set 2023-12-15T17:01:41.131 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd749ce7640 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:41.131 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd749ce7640 10 monclient: _renew_subs 2023-12-15T17:01:41.131 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd749ce7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:41.131 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd749ce7640 1 -- 172.21.15.69:0/858499531 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7441444a0 con 0x7fd74413f460 2023-12-15T17:01:41.131 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.125+0000 7fd749ce7640 1 librados: init done 2023-12-15T17:01:41.131 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.126+0000 7fd749ce7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:41.131 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.126+0000 7fd749ce7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:41.131 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.126+0000 7fd749ce7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:41.131 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.126+0000 7fd749ce7640 1 -- 172.21.15.69:0/858499531 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7440ad000 con 0x7fd74413f460 2023-12-15T17:01:41.575 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.574+0000 7fd72bfff640 1 -- 172.21.15.69:0/858499531 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 4) v1 ==== 89314+0+0 (secure 0 0 0) 0x7fd73804e470 con 0x7fd74413f460 2023-12-15T17:01:41.575 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.575+0000 7fd72bfff640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fd73006cb40 0x7fd73006f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:41.576 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.575+0000 7fd7437fe640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fd73006cb40 0x7fd73006f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:41.576 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.575+0000 7fd7437fe640 10 monclient: get_auth_request con 0x7fd73006cb40 auth_method 0 2023-12-15T17:01:41.577 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.576+0000 7fd7437fe640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fd73006cb40 0x7fd73006f000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd734009aa0 tx=0x7fd734009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:41.577 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.576+0000 7fd72bfff640 10 client.4199.objecter ms_handle_connect 0x7fd73006cb40 2023-12-15T17:01:41.599 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.598+0000 7fd72bfff640 1 -- 172.21.15.69:0/858499531 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd73807a1a0 con 0x7fd74413f460 2023-12-15T17:01:41.599 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.598+0000 7fd72bfff640 10 client.4199.objecter ms_dispatch 0x7fd744000b90 osd_map(10..10 src has 1..10) v4 2023-12-15T17:01:41.600 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.598+0000 7fd72bfff640 3 client.4199.objecter handle_osd_map got epochs [10,10] > 0 2023-12-15T17:01:41.600 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.598+0000 7fd72bfff640 3 client.4199.objecter handle_osd_map decoding full epoch 10 2023-12-15T17:01:41.600 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.598+0000 7fd72bfff640 20 client.4199.objecter dump_active .. 0 homeless 2023-12-15T17:01:41.600 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.598+0000 7fd72bfff640 1 -- 172.21.15.69:0/858499531 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fd7380764e0 con 0x7fd74413f460 2023-12-15T17:01:41.600 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.598+0000 7fd72bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:41.600 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.598+0000 7fd72bfff640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:41.911 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.910+0000 7fd749ce7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:41.911 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.910+0000 7fd749ce7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:41.911 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.910+0000 7fd749ce7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:41.911 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.910+0000 7fd749ce7640 1 -- 172.21.15.69:0/858499531 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7440ab010 con 0x7fd74413f460 2023-12-15T17:01:41.927 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.926+0000 7fd72bfff640 1 -- 172.21.15.69:0/858499531 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fd738079b00 con 0x7fd74413f460 2023-12-15T17:01:41.927 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.926+0000 7fd72bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:41.927 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.926+0000 7fd72bfff640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:41.927 INFO:teuthology.misc.health.smithi069.stdout: 2023-12-15T17:01:41.927 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"1a9f4b8c-183d-4435-93ef-5057f2b4aded","created":"2023-12-15T17:01:02.006221+0000","modified":"2023-12-15T17:01:08.340674+0000","last_up_change":"0.000000","last_in_change":"2023-12-15T17:01:08.340674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51cce62a-d177-4e4b-a814-7e537407f397","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1556db3a-b386-444f-8327-3dce2ad46d33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eaac40b5-9fc0-4e13-a322-763a9f3b685f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"71221e7e-8713-4fc4-bef1-c1cbe40627da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2c9feecc-d3b5-429d-98fe-e6eddc91eb8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac6e8a35-6df7-48d4-b59a-1f87eceaacfa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T17:01:41.930 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.928+0000 7fd749ce7640 10 librados: watch_flush enter 2023-12-15T17:01:41.930 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.928+0000 7fd749ce7640 10 librados: watch_flush exit 2023-12-15T17:01:41.930 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.928+0000 7fd749ce7640 20 client.4199.objecter shutdown clearing up homeless session... 2023-12-15T17:01:41.930 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.928+0000 7fd749ce7640 10 client.4199.objecter successfully canceled tick 2023-12-15T17:01:41.930 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.928+0000 7fd749ce7640 1 -- 172.21.15.69:0/858499531 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fd73006cb40 msgr2=0x7fd73006f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.928+0000 7fd749ce7640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fd73006cb40 0x7fd73006f000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd734009aa0 tx=0x7fd734009190 comp rx=0 tx=0).stop 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.928+0000 7fd749ce7640 10 monclient: shutdown 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.928+0000 7fd749ce7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.928+0000 7fd749ce7640 1 -- 172.21.15.69:0/858499531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd74413f460 msgr2=0x7fd7440ac930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.928+0000 7fd749ce7640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd74413f460 0x7fd7440ac930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd7380096f0 tx=0x7fd7380046c0 comp rx=0 tx=0).stop 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.929+0000 7fd749ce7640 1 -- 172.21.15.69:0/858499531 shutdown_connections 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.929+0000 7fd749ce7640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fd73006cb40 0x7fd73006f000 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.929+0000 7fd749ce7640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd744142710 0x7fd7441dabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.929+0000 7fd749ce7640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd74413f460 0x7fd7440ac930 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.929+0000 7fd749ce7640 1 --2- 172.21.15.69:0/858499531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7440a89e0 0x7fd7440abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.929+0000 7fd749ce7640 1 -- 172.21.15.69:0/858499531 >> 172.21.15.69:0/858499531 conn(0x7fd7440b7940 msgr2=0x7fd74413e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.929+0000 7fd749ce7640 1 -- 172.21.15.69:0/858499531 shutdown_connections 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.929+0000 7fd749ce7640 1 -- 172.21.15.69:0/858499531 wait complete. 2023-12-15T17:01:41.931 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:41.929+0000 7fd749ce7640 1 librados: shutdown 2023-12-15T17:01:41.940 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-15T17:01:43.921 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:01:43.863+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-12-15T17:01:44.465 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-12-15T17:01:44.459+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-12-15T17:01:44.544 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-12-15T17:01:44.540+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-12-15T17:01:44.694 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-12-15T17:01:44.690+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-12-15T17:01:44.749 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:01:44.745+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-12-15T17:01:44.979 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:01:44.976+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-12-15T17:01:45.293 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:01:45.290+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-12-15T17:01:45.638 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-12-15T17:01:45.635+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-12-15T17:01:47.309 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-12-15T17:01:47.305+0000 ac7d640 -1 osd.6 0 waiting for initial osdmap 2023-12-15T17:01:47.317 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-12-15T17:01:47.313+0000 c493640 -1 osd.5 0 waiting for initial osdmap 2023-12-15T17:01:47.335 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:01:47.332+0000 ac7d640 -1 osd.4 0 waiting for initial osdmap 2023-12-15T17:01:47.727 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-12-15T17:01:47.712+0000 12256640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-15T17:01:47.768 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:01:47.753+0000 12256640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-15T17:01:47.769 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-12-15T17:01:47.755+0000 12a57640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-15T17:01:48.351 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:01:48.347+0000 c493640 -1 osd.7 0 waiting for initial osdmap 2023-12-15T17:01:48.424 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:01:48.419+0000 c493640 -1 osd.0 0 waiting for initial osdmap 2023-12-15T17:01:48.456 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:01:48.452+0000 ac7d640 -1 osd.2 0 waiting for initial osdmap 2023-12-15T17:01:48.464 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-12-15T17:01:48.459+0000 c493640 -1 osd.3 0 waiting for initial osdmap 2023-12-15T17:01:48.532 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-12-15T17:01:48.527+0000 c493640 -1 osd.1 0 waiting for initial osdmap 2023-12-15T17:01:48.729 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:01:48.713+0000 12256640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-15T17:01:48.746 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-15T17:01:48.861 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-12-15T17:01:48.846+0000 12256640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-15T17:01:48.870 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.869+0000 7f2b28856640 10 monclient: get_monmap_and_config 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.869+0000 7f2b28856640 10 monclient: build_initial_monmap 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.869+0000 7f2b28856640 10 monclient: monmap: 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:01:48.870418+0000 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:01:48.870418+0000 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:48.871 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:48.872 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:48.872 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:48.872 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.870+0000 7f2b28856640 1 Processor -- start 2023-12-15T17:01:48.872 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.870+0000 7f2b28856640 1 -- start start 2023-12-15T17:01:48.872 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.870+0000 7f2b28856640 10 monclient: init 2023-12-15T17:01:48.872 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:48.872 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-12-15T17:01:48.873 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b240a8a00 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:48.873 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b24146150 addr v1:172.21.15.69:6789/0 2023-12-15T17:01:48.873 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2b240a8de0 0x7f2b2413f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:48.873 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 10 monclient(hunting): picked mon.noname-c con 0x7f2b240a8de0 addr v2:172.21.15.192:3300/0 2023-12-15T17:01:48.873 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b241455b0 con 0x7f2b24146150 2023-12-15T17:01:48.873 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b240aa970 con 0x7f2b240a8a00 2023-12-15T17:01:48.873 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:48.873 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b240aac50 con 0x7f2b240a8de0 2023-12-15T17:01:48.873 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:48.873 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.872+0000 7f2b28856640 10 monclient(hunting): authenticate will time out at 1413.187521s 2023-12-15T17:01:48.874 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.874+0000 7f2b2259c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f2b240a8a00 0x7f2b24145a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:43482/0 (socket says 172.21.15.69:43482) 2023-12-15T17:01:48.874 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.874+0000 7f2b2259c640 1 -- 172.21.15.69:0/1825364422 learned_addr learned my addr 172.21.15.69:0/1825364422 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:01:48.875 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.875+0000 7f2b22d9d640 1 --2- 172.21.15.69:0/1825364422 >> v2:172.21.15.192:3300/0 conn(0x7f2b240a8de0 0x7f2b2413f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:48.875 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.875+0000 7f2b22d9d640 10 monclient(hunting): get_auth_request con 0x7f2b240a8de0 auth_method 0 2023-12-15T17:01:48.875 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.875+0000 7f2b22d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:48.875 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.875+0000 7f2b22d9d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:48.875 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.875+0000 7f2b22d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:48.877 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.876+0000 7f2b22d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:48.877 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.876+0000 7f2b22d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:48.877 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.876+0000 7f2b22d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:48.877 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:01:48.860+0000 12256640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-15T17:01:48.880 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.878+0000 7f2b22d9d640 10 monclient(hunting): handle_auth_done global_id 4249 payload 274 2023-12-15T17:01:48.880 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.878+0000 7f2b22d9d640 1 -- 172.21.15.69:0/1825364422 >> v1:172.21.15.192:6789/0 conn(0x7f2b240a8a00 legacy=0x7f2b24145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:48.880 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.879+0000 7f2b22d9d640 1 -- 172.21.15.69:0/1825364422 >> v1:172.21.15.69:6789/0 conn(0x7f2b24146150 legacy=0x7f2b2413eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:48.880 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.879+0000 7f2b22d9d640 10 monclient: _finish_hunting 0 2023-12-15T17:01:48.880 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.879+0000 7f2b22d9d640 1 monclient: found mon.noname-c 2023-12-15T17:01:48.880 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.879+0000 7f2b22d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:48.880 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.879+0000 7f2b22d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:01:48.880 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.879+0000 7f2b22d9d640 1 -- 172.21.15.69:0/1825364422 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b240aaf30 con 0x7f2b240a8de0 2023-12-15T17:01:48.882 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:01:48.861+0000 12256640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-15T17:01:48.883 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.882+0000 7f2b22d9d640 1 --2- 172.21.15.69:0/1825364422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b240a8de0 0x7f2b2413f3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009d90 tx=0x7f2b1c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6f70827ac974a5f server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:48.887 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 1 -- 172.21.15.69:0/1825364422 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c02cd00 con 0x7f2b240a8de0 2023-12-15T17:01:48.887 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:48.887 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:01:48.887 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 10 monclient: dump: 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 10 monclient: _finish_auth 0 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 10 monclient: _check_auth_tickets 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 1 -- 172.21.15.69:0/1825364422 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1c02ce60 con 0x7f2b240a8de0 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:48.888 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 1 -- 172.21.15.69:0/1825364422 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c034660 con 0x7f2b240a8de0 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.886+0000 7f2b2159a640 10 monclient: dump: 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:48.889 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:48.890 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.888+0000 7f2b28856640 5 monclient: authenticate success, global_id 4249 2023-12-15T17:01:48.890 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.888+0000 7f2b28856640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:48.890 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.888+0000 7f2b28856640 4 set_mon_vals no callback set 2023-12-15T17:01:48.890 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.889+0000 7f2b28856640 10 monclient: shutdown 2023-12-15T17:01:48.890 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.889+0000 7f2b28856640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:48.890 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.889+0000 7f2b28856640 1 -- 172.21.15.69:0/1825364422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b240a8de0 msgr2=0x7f2b2413f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:48.890 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.889+0000 7f2b28856640 1 --2- 172.21.15.69:0/1825364422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b240a8de0 0x7f2b2413f3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009d90 tx=0x7f2b1c02bf50 comp rx=0 tx=0).stop 2023-12-15T17:01:48.891 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.890+0000 7f2b28856640 1 -- 172.21.15.69:0/1825364422 shutdown_connections 2023-12-15T17:01:48.891 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.890+0000 7f2b28856640 1 --2- 172.21.15.69:0/1825364422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b240a8de0 0x7f2b2413f3b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:48.891 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.890+0000 7f2b28856640 1 -- 172.21.15.69:0/1825364422 >> 172.21.15.69:0/1825364422 conn(0x7f2b24066e80 msgr2=0x7f2b240b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:48.891 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.890+0000 7f2b28856640 1 -- 172.21.15.69:0/1825364422 shutdown_connections 2023-12-15T17:01:48.891 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.890+0000 7f2b28856640 1 -- 172.21.15.69:0/1825364422 wait complete. 2023-12-15T17:01:48.892 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.891+0000 7f2b28856640 10 monclient: build_initial_monmap 2023-12-15T17:01:48.892 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.891+0000 7f2b28856640 10 monclient: monmap: 2023-12-15T17:01:48.892 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:48.892 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 librados: starting msgr at 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 librados: starting objecter 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 Processor -- start 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 -- start start 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 librados: setting wanted keys 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 librados: calling monclient init 2023-12-15T17:01:48.893 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 10 monclient: init 2023-12-15T17:01:48.895 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:48.895 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:01:48.895 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b240a8a00 0x7f2b24143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:48.895 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 10 monclient(hunting): picked mon.noname-c con 0x7f2b240a8a00 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:48.895 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b240a8de0 0x7f2b2413aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:48.895 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 10 monclient(hunting): picked mon.noname-b con 0x7f2b240a8de0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:48.895 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b24146150 0x7f2b2413af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:48.895 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 10 monclient(hunting): picked mon.noname-a con 0x7f2b24146150 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b24142a10 con 0x7f2b24146150 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b24144270 con 0x7f2b240a8de0 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2413bee0 con 0x7f2b240a8a00 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.892+0000 7f2b28856640 10 monclient(hunting): authenticate will time out at 1413.208244s 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b2259c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b240a8a00 0x7f2b24143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b21d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b240a8de0 0x7f2b2413aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b21d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b240a8de0 0x7f2b2413aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:58474/0 (socket says 172.21.15.69:58474) 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b21d9b640 1 -- 172.21.15.69:0/674023333 learned_addr learned my addr 172.21.15.69:0/674023333 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b21d9b640 10 monclient(hunting): get_auth_request con 0x7f2b240a8de0 auth_method 0 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b21d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:48.896 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b21d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:48.897 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b21d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:48.897 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b2259c640 10 monclient(hunting): get_auth_request con 0x7f2b240a8a00 auth_method 0 2023-12-15T17:01:48.897 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b2259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:48.897 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b2259c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:48.897 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.893+0000 7f2b2259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:48.897 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.894+0000 7f2b21d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:48.897 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.894+0000 7f2b21d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:48.897 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.894+0000 7f2b21d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:48.897 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.894+0000 7f2b2259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:48.897 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.894+0000 7f2b2259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:48.897 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.894+0000 7f2b2259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:48.898 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.896+0000 7f2b21d9b640 10 monclient(hunting): handle_auth_done global_id 4252 payload 824 2023-12-15T17:01:48.898 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.896+0000 7f2b21d9b640 1 -- 172.21.15.69:0/674023333 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b240a8a00 msgr2=0x7f2b24143660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:48.899 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.896+0000 7f2b21d9b640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b240a8a00 0x7f2b24143660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:48.899 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.896+0000 7f2b21d9b640 1 -- 172.21.15.69:0/674023333 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b24146150 msgr2=0x7f2b2413af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:48.899 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.896+0000 7f2b21d9b640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b24146150 0x7f2b2413af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:48.899 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.896+0000 7f2b21d9b640 10 monclient: _finish_hunting 0 2023-12-15T17:01:48.899 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.896+0000 7f2b21d9b640 1 monclient: found mon.noname-b 2023-12-15T17:01:48.899 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.897+0000 7f2b21d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:48.899 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.897+0000 7f2b21d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:01:48.899 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.897+0000 7f2b21d9b640 1 -- 172.21.15.69:0/674023333 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c009a70 con 0x7f2b240a8de0 2023-12-15T17:01:48.900 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.900+0000 7f2b21d9b640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b240a8de0 0x7f2b2413aa50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800c430 tx=0x7f2b1800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:48.901 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.900+0000 7f2b12ffd640 10 client.?.objecter ms_handle_connect 0x7f2b240a8de0 2023-12-15T17:01:48.901 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.900+0000 7f2b12ffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b12ffd640 1 -- 172.21.15.69:0/674023333 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b180052d0 con 0x7f2b240a8de0 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b12ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b12ffd640 10 monclient: dump: 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:48.975 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b12ffd640 10 monclient: _finish_auth 0 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b12ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b12ffd640 1 -- 172.21.15.69:0/674023333 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b18005430 con 0x7f2b240a8de0 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b28856640 5 monclient: authenticate success, global_id 4252 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b12ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b12ffd640 1 -- 172.21.15.69:0/674023333 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18003830 con 0x7f2b240a8de0 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b28856640 10 monclient: _renew_subs 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b28856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b28856640 1 -- 172.21.15.69:0/674023333 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2413fd20 con 0x7f2b240a8de0 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b28856640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b28856640 10 monclient: _renew_subs 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b28856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b20d99640 4 set_mon_vals no callback set 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b28856640 1 -- 172.21.15.69:0/674023333 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2413b6b0 con 0x7f2b240a8de0 2023-12-15T17:01:48.976 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.974+0000 7f2b28856640 1 librados: init done 2023-12-15T17:01:48.977 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.975+0000 7f2b12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:48.977 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.975+0000 7f2b12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:48.977 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.975+0000 7f2b12ffd640 10 monclient: dump: 2023-12-15T17:01:48.977 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:48.977 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:48.977 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:48.978 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:48.978 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:48.978 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:48.978 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:48.978 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:48.978 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:48.978 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:48.978 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.975+0000 7f2b28856640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:48.978 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.975+0000 7f2b28856640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:48.978 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.975+0000 7f2b28856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:48.978 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:48.975+0000 7f2b28856640 1 -- 172.21.15.69:0/674023333 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b24069c00 con 0x7f2b240a8de0 2023-12-15T17:01:48.991 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-12-15T17:01:48.973+0000 12256640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-15T17:01:49.221 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.220+0000 7f2b12ffd640 1 -- 172.21.15.69:0/674023333 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 4) v1 ==== 89314+0+0 (secure 0 0 0) 0x7f2b18003990 con 0x7f2b240a8de0 2023-12-15T17:01:49.221 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.220+0000 7f2b12ffd640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2b0406cb10 0x7f2b0406efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:49.221 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.220+0000 7f2b2259c640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2b0406cb10 0x7f2b0406efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:49.221 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.220+0000 7f2b2259c640 10 monclient: get_auth_request con 0x7f2b0406cb10 auth_method 0 2023-12-15T17:01:49.222 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.221+0000 7f2b12ffd640 10 client.4252.objecter ms_handle_connect 0x7f2b0406cb10 2023-12-15T17:01:49.222 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.221+0000 7f2b2259c640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2b0406cb10 0x7f2b0406efd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2b14005880 tx=0x7f2b14009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:49.235 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.234+0000 7f2b12ffd640 1 -- 172.21.15.69:0/674023333 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f2b18057d80 con 0x7f2b240a8de0 2023-12-15T17:01:49.235 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.234+0000 7f2b12ffd640 10 client.4252.objecter ms_dispatch 0x7f2b24000b90 osd_map(13..13 src has 1..13) v4 2023-12-15T17:01:49.235 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.234+0000 7f2b12ffd640 3 client.4252.objecter handle_osd_map got epochs [13,13] > 0 2023-12-15T17:01:49.235 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.234+0000 7f2b12ffd640 3 client.4252.objecter handle_osd_map decoding full epoch 13 2023-12-15T17:01:49.235 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.234+0000 7f2b12ffd640 20 client.4252.objecter dump_active .. 0 homeless 2023-12-15T17:01:49.235 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.234+0000 7f2b12ffd640 1 -- 172.21.15.69:0/674023333 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2b180540c0 con 0x7f2b240a8de0 2023-12-15T17:01:49.236 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.234+0000 7f2b12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:49.236 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.234+0000 7f2b12ffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:49.443 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.442+0000 7f2b28856640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:49.443 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.442+0000 7f2b28856640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:49.443 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.442+0000 7f2b28856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:49.443 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.442+0000 7f2b28856640 1 -- 172.21.15.69:0/674023333 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b24142c50 con 0x7f2b240a8de0 2023-12-15T17:01:49.480 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.479+0000 7f2b12ffd640 1 -- 172.21.15.69:0/674023333 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+9025 (secure 0 0 0) 0x7f2b180576e0 con 0x7f2b240a8de0 2023-12-15T17:01:49.481 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.479+0000 7f2b12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:49.481 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.479+0000 7f2b12ffd640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:49.481 INFO:teuthology.misc.health.smithi069.stdout: 2023-12-15T17:01:49.481 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":14,"fsid":"1a9f4b8c-183d-4435-93ef-5057f2b4aded","created":"2023-12-15T17:01:02.006221+0000","modified":"2023-12-15T17:01:49.018992+0000","last_up_change":"2023-12-15T17:01:49.018992+0000","last_in_change":"2023-12-15T17:01:08.340674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51cce62a-d177-4e4b-a814-7e537407f397","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1556db3a-b386-444f-8327-3dce2ad46d33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eaac40b5-9fc0-4e13-a322-763a9f3b685f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"71221e7e-8713-4fc4-bef1-c1cbe40627da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6803","nonce":1420550374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6805","nonce":1420550374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6809","nonce":1420550374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6807","nonce":1420550374}]},"public_addr":"172.21.15.192:6803/1420550374","cluster_addr":"172.21.15.192:6805/1420550374","heartbeat_back_addr":"172.21.15.192:6809/1420550374","heartbeat_front_addr":"172.21.15.192:6807/1420550374","state":["exists","up"]},{"osd":5,"uuid":"2c9feecc-d3b5-429d-98fe-e6eddc91eb8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6811","nonce":2003653981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6813","nonce":2003653981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6817","nonce":2003653981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6815","nonce":2003653981}]},"public_addr":"172.21.15.192:6811/2003653981","cluster_addr":"172.21.15.192:6813/2003653981","heartbeat_back_addr":"172.21.15.192:6817/2003653981","heartbeat_front_addr":"172.21.15.192:6815/2003653981","state":["exists","up"]},{"osd":6,"uuid":"5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6819","nonce":2563835645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6821","nonce":2563835645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6825","nonce":2563835645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6823","nonce":2563835645}]},"public_addr":"172.21.15.192:6819/2563835645","cluster_addr":"172.21.15.192:6821/2563835645","heartbeat_back_addr":"172.21.15.192:6825/2563835645","heartbeat_front_addr":"172.21.15.192:6823/2563835645","state":["exists","up"]},{"osd":7,"uuid":"ac6e8a35-6df7-48d4-b59a-1f87eceaacfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6827","nonce":4283271495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6829","nonce":4283271495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6833","nonce":4283271495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6831","nonce":4283271495}]},"public_addr":"172.21.15.192:6827/4283271495","cluster_addr":"172.21.15.192:6829/4283271495","heartbeat_back_addr":"172.21.15.192:6833/4283271495","heartbeat_front_addr":"172.21.15.192:6831/4283271495","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T17:01:49.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 10 librados: watch_flush enter 2023-12-15T17:01:49.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 10 librados: watch_flush exit 2023-12-15T17:01:49.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 20 client.4252.objecter shutdown clearing up homeless session... 2023-12-15T17:01:49.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 10 client.4252.objecter successfully canceled tick 2023-12-15T17:01:49.483 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 -- 172.21.15.69:0/674023333 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2b0406cb10 msgr2=0x7f2b0406efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2b0406cb10 0x7f2b0406efd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2b14005880 tx=0x7f2b14009040 comp rx=0 tx=0).stop 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 10 monclient: shutdown 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 -- 172.21.15.69:0/674023333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b240a8de0 msgr2=0x7f2b2413aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b240a8de0 0x7f2b2413aa50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800c430 tx=0x7f2b1800c8f0 comp rx=0 tx=0).stop 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 -- 172.21.15.69:0/674023333 shutdown_connections 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2b0406cb10 0x7f2b0406efd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b24146150 0x7f2b2413af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b240a8de0 0x7f2b2413aa50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 --2- 172.21.15.69:0/674023333 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b240a8a00 0x7f2b24143660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 -- 172.21.15.69:0/674023333 >> 172.21.15.69:0/674023333 conn(0x7f2b24066e80 msgr2=0x7f2b241449c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 -- 172.21.15.69:0/674023333 shutdown_connections 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 -- 172.21.15.69:0/674023333 wait complete. 2023-12-15T17:01:49.484 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:49.482+0000 7f2b28856640 1 librados: shutdown 2023-12-15T17:01:49.492 DEBUG:teuthology.misc:4 of 8 OSDs are up 2023-12-15T17:01:56.299 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-15T17:01:56.415 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.414+0000 7f2252244640 10 monclient: get_monmap_and_config 2023-12-15T17:01:56.415 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.414+0000 7f2252244640 10 monclient: build_initial_monmap 2023-12-15T17:01:56.415 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.414+0000 7f2252244640 10 monclient: monmap: 2023-12-15T17:01:56.415 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:56.415 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:56.415 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:01:56.414818+0000 2023-12-15T17:01:56.415 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:01:56.414818+0000 2023-12-15T17:01:56.415 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:56.415 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:56.415 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:56.415 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:56.416 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:56.416 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:56.416 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:56.416 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:56.418 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 1 Processor -- start 2023-12-15T17:01:56.418 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 1 -- start start 2023-12-15T17:01:56.418 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 10 monclient: init 2023-12-15T17:01:56.418 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:56.418 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-12-15T17:01:56.418 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f224c0a89e0 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:56.418 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c13f630 0x7f224c1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:56.418 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 10 monclient(hunting): picked mon.noname-b con 0x7f224c13f630 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f224c142710 0x7f224c14c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 10 monclient(hunting): picked mon.noname-a con 0x7f224c142710 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224c0aa770 con 0x7f224c0a89e0 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224c0aa9c0 con 0x7f224c142710 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0aaca0 con 0x7f224c13f630 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.415+0000 7f2252244640 10 monclient(hunting): authenticate will time out at 1420.731253s 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.416+0000 7f224affd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c13f630 0x7f224c1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.416+0000 7f224bfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f224c142710 0x7f224c14c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.416+0000 7f224b7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f224c0a89e0 0x7f224c13ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:43494/0 (socket says 172.21.15.69:43494) 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.416+0000 7f224b7fe640 1 -- 172.21.15.69:0/866798741 learned_addr learned my addr 172.21.15.69:0/866798741 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.416+0000 7f224affd640 10 monclient(hunting): get_auth_request con 0x7f224c13f630 auth_method 0 2023-12-15T17:01:56.419 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.416+0000 7f224affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.416+0000 7f224affd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.416+0000 7f224affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.417+0000 7f224bfff640 10 monclient(hunting): get_auth_request con 0x7f224c142710 auth_method 0 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.417+0000 7f224bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.417+0000 7f224bfff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.417+0000 7f224bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.417+0000 7f224affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.417+0000 7f224affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.417+0000 7f224affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.417+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.417+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:56.420 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.417+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:56.421 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.419+0000 7f224a7fc640 1 -- 172.21.15.69:0/866798741 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891258350 0 0) 0x7f224c0aa770 con 0x7f224c0a89e0 2023-12-15T17:01:56.421 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.419+0000 7f224a7fc640 10 monclient(hunting): _negotiate 2023-12-15T17:01:56.421 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.419+0000 7f224a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:56.421 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.419+0000 7f224a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:56.421 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.419+0000 7f224a7fc640 10 monclient(hunting): my global_id is 4261 2023-12-15T17:01:56.422 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.419+0000 7f224a7fc640 1 -- 172.21.15.69:0/866798741 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2238003520 con 0x7f224c0a89e0 2023-12-15T17:01:56.422 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.420+0000 7f224bfff640 10 monclient(hunting): handle_auth_done global_id 4239 payload 274 2023-12-15T17:01:56.422 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.420+0000 7f224bfff640 1 -- 172.21.15.69:0/866798741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c13f630 msgr2=0x7f224c1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:56.422 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.420+0000 7f224bfff640 1 --2- 172.21.15.69:0/866798741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c13f630 0x7f224c1421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:56.422 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.420+0000 7f224bfff640 1 -- 172.21.15.69:0/866798741 >> v1:172.21.15.192:6789/0 conn(0x7f224c0a89e0 legacy=0x7f224c13ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:56.422 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.420+0000 7f224bfff640 10 monclient: _finish_hunting 0 2023-12-15T17:01:56.422 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.420+0000 7f224bfff640 1 monclient: found mon.noname-a 2023-12-15T17:01:56.422 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.420+0000 7f224bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:56.422 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.420+0000 7f224bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:01:56.422 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.420+0000 7f224bfff640 1 -- 172.21.15.69:0/866798741 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c0aaf80 con 0x7f224c142710 2023-12-15T17:01:56.422 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.420+0000 7f224affd640 1 --2- 172.21.15.69:0/866798741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c13f630 0x7f224c1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-15T17:01:56.423 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.422+0000 7f224bfff640 1 --2- 172.21.15.69:0/866798741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f224c142710 0x7f224c14c800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f22440097a0 tx=0x7f224402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbce2cfe6aa29e9c server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:56.427 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 1 -- 172.21.15.69:0/866798741 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2244031780 con 0x7f224c142710 2023-12-15T17:01:56.427 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:56.427 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:01:56.427 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 10 monclient: dump: 2023-12-15T17:01:56.427 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:56.427 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:56.427 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:56.427 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:56.427 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:56.427 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 10 monclient: _finish_auth 0 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 10 monclient: _check_auth_tickets 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 1 -- 172.21.15.69:0/866798741 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22440318e0 con 0x7f224c142710 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 1 -- 172.21.15.69:0/866798741 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2244031c70 con 0x7f224c142710 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f2252244640 5 monclient: authenticate success, global_id 4239 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f224a7fc640 10 monclient: dump: 2023-12-15T17:01:56.428 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:56.429 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:56.429 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:56.429 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:56.429 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:56.429 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:56.429 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:56.429 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:56.429 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:56.429 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:56.429 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f2252244640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:56.429 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.426+0000 7f2252244640 4 set_mon_vals no callback set 2023-12-15T17:01:56.432 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.427+0000 7f2252244640 10 monclient: shutdown 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.427+0000 7f2252244640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.427+0000 7f2252244640 1 -- 172.21.15.69:0/866798741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f224c142710 msgr2=0x7f224c14c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.427+0000 7f2252244640 1 --2- 172.21.15.69:0/866798741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f224c142710 0x7f224c14c800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f22440097a0 tx=0x7f224402b820 comp rx=0 tx=0).stop 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.427+0000 7f2252244640 1 -- 172.21.15.69:0/866798741 shutdown_connections 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.427+0000 7f2252244640 1 --2- 172.21.15.69:0/866798741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f224c142710 0x7f224c14c800 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.427+0000 7f2252244640 1 --2- 172.21.15.69:0/866798741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c13f630 0x7f224c1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.427+0000 7f2252244640 1 -- 172.21.15.69:0/866798741 >> 172.21.15.69:0/866798741 conn(0x7f224c0b7940 msgr2=0x7f224c0b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.427+0000 7f2252244640 1 -- 172.21.15.69:0/866798741 shutdown_connections 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.427+0000 7f2252244640 1 -- 172.21.15.69:0/866798741 wait complete. 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.428+0000 7f2252244640 10 monclient: build_initial_monmap 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.428+0000 7f2252244640 10 monclient: monmap: 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:56.433 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.428+0000 7f2252244640 1 librados: starting msgr at 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.428+0000 7f2252244640 1 librados: starting objecter 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.428+0000 7f2252244640 1 Processor -- start 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 1 -- start start 2023-12-15T17:01:56.434 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 1 librados: setting wanted keys 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 1 librados: calling monclient init 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 10 monclient: init 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f224c0a89e0 0x7f224c0ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 10 monclient(hunting): picked mon.noname-b con 0x7f224c0a89e0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f224c13f630 0x7f224c1d6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 10 monclient(hunting): picked mon.noname-a con 0x7f224c13f630 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c142710 0x7f224c1d7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 10 monclient(hunting): picked mon.noname-c con 0x7f224c142710 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c144a40 con 0x7f224c13f630 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.429+0000 7f2252244640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f2252244640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c1460d0 con 0x7f224c0a89e0 2023-12-15T17:01:56.435 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f2252244640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f2252244640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c142fb0 con 0x7f224c142710 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f2252244640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f2252244640 10 monclient(hunting): authenticate will time out at 1420.745385s 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f224affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f224c13f630 0x7f224c1d6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f224bfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c142710 0x7f224c1d7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f224b7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f224c0a89e0 0x7f224c0ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f224bfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c142710 0x7f224c1d7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36508/0 (socket says 172.21.15.69:36508) 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f224bfff640 1 -- 172.21.15.69:0/3370221989 learned_addr learned my addr 172.21.15.69:0/3370221989 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f224bfff640 10 monclient(hunting): get_auth_request con 0x7f224c142710 auth_method 0 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f224bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f224bfff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.430+0000 7f224bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224b7fe640 10 monclient(hunting): get_auth_request con 0x7f224c0a89e0 auth_method 0 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:56.436 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224affd640 10 monclient(hunting): get_auth_request con 0x7f224c13f630 auth_method 0 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224affd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:56.437 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.431+0000 7f224affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:56.438 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.434+0000 7f224b7fe640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2023-12-15T17:01:56.438 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.434+0000 7f224b7fe640 1 -- 172.21.15.69:0/3370221989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c142710 msgr2=0x7f224c1d7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:56.438 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.434+0000 7f224b7fe640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c142710 0x7f224c1d7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:56.439 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.434+0000 7f224b7fe640 1 -- 172.21.15.69:0/3370221989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f224c13f630 msgr2=0x7f224c1d6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:56.439 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.434+0000 7f224b7fe640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f224c13f630 0x7f224c1d6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:56.439 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.434+0000 7f224b7fe640 10 monclient: _finish_hunting 0 2023-12-15T17:01:56.439 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.434+0000 7f224b7fe640 1 monclient: found mon.noname-b 2023-12-15T17:01:56.439 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.434+0000 7f224b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:56.439 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.434+0000 7f224b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:01:56.439 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.434+0000 7f224b7fe640 1 -- 172.21.15.69:0/3370221989 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2234009400 con 0x7f224c0a89e0 2023-12-15T17:01:56.439 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.436+0000 7f224b7fe640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f224c0a89e0 0x7f224c0ab9d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f223c00cbc0 tx=0x7f223c00a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:56.439 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.436+0000 7f2243fff640 10 client.?.objecter ms_handle_connect 0x7f224c0a89e0 2023-12-15T17:01:56.439 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.436+0000 7f2243fff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:56.441 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2243fff640 1 -- 172.21.15.69:0/3370221989 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c00aca0 con 0x7f224c0a89e0 2023-12-15T17:01:56.441 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2243fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:56.441 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2243fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:56.441 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2243fff640 10 monclient: dump: 2023-12-15T17:01:56.441 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:56.441 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:56.441 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:56.441 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:56.441 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:56.441 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:56.441 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2243fff640 10 monclient: _finish_auth 0 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2243fff640 10 monclient: _check_auth_tickets 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2243fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2243fff640 1 -- 172.21.15.69:0/3370221989 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223c00ae00 con 0x7f224c0a89e0 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2252244640 5 monclient: authenticate success, global_id 4264 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2252244640 10 monclient: _renew_subs 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2252244640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2252244640 1 -- 172.21.15.69:0/3370221989 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2244009400 con 0x7f224c0a89e0 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.439+0000 7f2243fff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.440+0000 7f2243fff640 1 -- 172.21.15.69:0/3370221989 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c012600 con 0x7f224c0a89e0 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.440+0000 7f2243fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:56.442 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.440+0000 7f2243fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.440+0000 7f2243fff640 10 monclient: dump: 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr: 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.440+0000 7f2249ffb640 4 set_mon_vals no callback set 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.440+0000 7f2252244640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.440+0000 7f2252244640 10 monclient: _renew_subs 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.440+0000 7f2252244640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:56.443 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.440+0000 7f2252244640 1 -- 172.21.15.69:0/3370221989 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c1d7960 con 0x7f224c0a89e0 2023-12-15T17:01:56.444 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.440+0000 7f2252244640 1 librados: init done 2023-12-15T17:01:56.444 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.441+0000 7f2252244640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:56.444 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.441+0000 7f2252244640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:56.444 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.441+0000 7f2252244640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:56.444 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.441+0000 7f2252244640 1 -- 172.21.15.69:0/3370221989 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224c1d7bb0 con 0x7f224c0a89e0 2023-12-15T17:01:56.458 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.457+0000 7f2243fff640 1 -- 172.21.15.69:0/3370221989 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f223c007070 con 0x7f224c0a89e0 2023-12-15T17:01:56.459 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.458+0000 7f2243fff640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f222c06cba0 0x7f222c06f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:56.459 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.458+0000 7f224affd640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f222c06cba0 0x7f222c06f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:56.459 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.458+0000 7f224affd640 10 monclient: get_auth_request con 0x7f222c06cba0 auth_method 0 2023-12-15T17:01:56.459 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.459+0000 7f224affd640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f222c06cba0 0x7f222c06f060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f224c145750 tx=0x7f2234009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:56.459 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.459+0000 7f2243fff640 10 client.4264.objecter ms_handle_connect 0x7f222c06cba0 2023-12-15T17:01:56.683 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.682+0000 7f2243fff640 1 -- 172.21.15.69:0/3370221989 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f223c0588c0 con 0x7f224c0a89e0 2023-12-15T17:01:56.683 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.682+0000 7f2243fff640 10 client.4264.objecter ms_dispatch 0x7f224c000b90 osd_map(18..18 src has 1..18) v4 2023-12-15T17:01:56.683 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.682+0000 7f2243fff640 3 client.4264.objecter handle_osd_map got epochs [18,18] > 0 2023-12-15T17:01:56.683 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.682+0000 7f2243fff640 3 client.4264.objecter handle_osd_map decoding full epoch 18 2023-12-15T17:01:56.683 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.682+0000 7f2243fff640 20 client.4264.objecter dump_active .. 0 homeless 2023-12-15T17:01:56.692 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.691+0000 7f2243fff640 1 -- 172.21.15.69:0/3370221989 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f223c054c00 con 0x7f224c0a89e0 2023-12-15T17:01:56.692 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.691+0000 7f2243fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:56.692 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.691+0000 7f2243fff640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:56.864 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.863+0000 7f2252244640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:56.864 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.863+0000 7f2252244640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:56.864 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.863+0000 7f2252244640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:56.864 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.863+0000 7f2252244640 1 -- 172.21.15.69:0/3370221989 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f224c0acc00 con 0x7f224c0a89e0 2023-12-15T17:01:56.949 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.948+0000 7f2243fff640 1 -- 172.21.15.69:0/3370221989 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12839 (secure 0 0 0) 0x7f223c017090 con 0x7f224c0a89e0 2023-12-15T17:01:56.950 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.948+0000 7f2243fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:01:56.950 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.948+0000 7f2243fff640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:01:56.950 INFO:teuthology.misc.health.smithi069.stdout: 2023-12-15T17:01:56.950 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":18,"fsid":"1a9f4b8c-183d-4435-93ef-5057f2b4aded","created":"2023-12-15T17:01:02.006221+0000","modified":"2023-12-15T17:01:53.790922+0000","last_up_change":"2023-12-15T17:01:50.252516+0000","last_in_change":"2023-12-15T17:01:08.340674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-15T17:01:49.711473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51cce62a-d177-4e4b-a814-7e537407f397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6817","nonce":541068705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6819","nonce":541068705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6823","nonce":541068705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6821","nonce":541068705}]},"public_addr":"172.21.15.69:6817/541068705","cluster_addr":"172.21.15.69:6819/541068705","heartbeat_back_addr":"172.21.15.69:6823/541068705","heartbeat_front_addr":"172.21.15.69:6821/541068705","state":["exists","up"]},{"osd":1,"uuid":"1556db3a-b386-444f-8327-3dce2ad46d33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6809","nonce":1683202761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6811","nonce":1683202761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6815","nonce":1683202761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6813","nonce":1683202761}]},"public_addr":"172.21.15.69:6809/1683202761","cluster_addr":"172.21.15.69:6811/1683202761","heartbeat_back_addr":"172.21.15.69:6815/1683202761","heartbeat_front_addr":"172.21.15.69:6813/1683202761","state":["exists","up"]},{"osd":2,"uuid":"eaac40b5-9fc0-4e13-a322-763a9f3b685f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6801","nonce":3322029776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6803","nonce":3322029776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6807","nonce":3322029776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6805","nonce":3322029776}]},"public_addr":"172.21.15.69:6801/3322029776","cluster_addr":"172.21.15.69:6803/3322029776","heartbeat_back_addr":"172.21.15.69:6807/3322029776","heartbeat_front_addr":"172.21.15.69:6805/3322029776","state":["exists","up"]},{"osd":3,"uuid":"5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6825","nonce":882900488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6827","nonce":882900488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6831","nonce":882900488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6829","nonce":882900488}]},"public_addr":"172.21.15.69:6825/882900488","cluster_addr":"172.21.15.69:6827/882900488","heartbeat_back_addr":"172.21.15.69:6831/882900488","heartbeat_front_addr":"172.21.15.69:6829/882900488","state":["exists","up"]},{"osd":4,"uuid":"71221e7e-8713-4fc4-bef1-c1cbe40627da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6803","nonce":1420550374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6805","nonce":1420550374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6809","nonce":1420550374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6807","nonce":1420550374}]},"public_addr":"172.21.15.192:6803/1420550374","cluster_addr":"172.21.15.192:6805/1420550374","heartbeat_back_addr":"172.21.15.192:6809/1420550374","heartbeat_front_addr":"172.21.15.192:6807/1420550374","state":["exists","up"]},{"osd":5,"uuid":"2c9feecc-d3b5-429d-98fe-e6eddc91eb8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6811","nonce":2003653981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6813","nonce":2003653981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6817","nonce":2003653981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6815","nonce":2003653981}]},"public_addr":"172.21.15.192:6811/2003653981","cluster_addr":"172.21.15.192:6813/2003653981","heartbeat_back_addr":"172.21.15.192:6817/2003653981","heartbeat_front_addr":"172.21.15.192:6815/2003653981","state":["exists","up"]},{"osd":6,"uuid":"5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6819","nonce":2563835645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6821","nonce":2563835645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6825","nonce":2563835645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6823","nonce":2563835645}]},"public_addr":"172.21.15.192:6819/2563835645","cluster_addr":"172.21.15.192:6821/2563835645","heartbeat_back_addr":"172.21.15.192:6825/2563835645","heartbeat_front_addr":"172.21.15.192:6823/2563835645","state":["exists","up"]},{"osd":7,"uuid":"ac6e8a35-6df7-48d4-b59a-1f87eceaacfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6827","nonce":4283271495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6829","nonce":4283271495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6833","nonce":4283271495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6831","nonce":4283271495}]},"public_addr":"172.21.15.192:6827/4283271495","cluster_addr":"172.21.15.192:6829/4283271495","heartbeat_back_addr":"172.21.15.192:6833/4283271495","heartbeat_front_addr":"172.21.15.192:6831/4283271495","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.990860+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.734395+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:47.257292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:47.560872+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.054415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.592183+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.562550+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.670431+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T17:01:56.952 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.950+0000 7f2252244640 10 librados: watch_flush enter 2023-12-15T17:01:56.952 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.950+0000 7f2252244640 10 librados: watch_flush exit 2023-12-15T17:01:56.952 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.950+0000 7f2252244640 20 client.4264.objecter shutdown clearing up homeless session... 2023-12-15T17:01:56.952 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.950+0000 7f2252244640 10 client.4264.objecter successfully canceled tick 2023-12-15T17:01:56.952 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.950+0000 7f2252244640 1 -- 172.21.15.69:0/3370221989 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f222c06cba0 msgr2=0x7f222c06f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:56.952 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f222c06cba0 0x7f222c06f060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f224c145750 tx=0x7f2234009190 comp rx=0 tx=0).stop 2023-12-15T17:01:56.952 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 10 monclient: shutdown 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 -- 172.21.15.69:0/3370221989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f224c0a89e0 msgr2=0x7f224c0ab9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f224c0a89e0 0x7f224c0ab9d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f223c00cbc0 tx=0x7f223c00a060 comp rx=0 tx=0).stop 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 -- 172.21.15.69:0/3370221989 shutdown_connections 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f222c06cba0 0x7f222c06f060 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f224c142710 0x7f224c1d7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f224c13f630 0x7f224c1d6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 --2- 172.21.15.69:0/3370221989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f224c0a89e0 0x7f224c0ab9d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 -- 172.21.15.69:0/3370221989 >> 172.21.15.69:0/3370221989 conn(0x7f224c0b7940 msgr2=0x7f224c0b2240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 -- 172.21.15.69:0/3370221989 shutdown_connections 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 -- 172.21.15.69:0/3370221989 wait complete. 2023-12-15T17:01:56.953 INFO:teuthology.misc.health.smithi069.stderr:2023-12-15T17:01:56.951+0000 7f2252244640 1 librados: shutdown 2023-12-15T17:01:56.962 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-15T17:01:56.962 INFO:tasks.ceph:Creating RBD pool 2023-12-15T17:01:56.963 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-15T17:01:57.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.064+0000 7f403156c640 10 monclient: get_monmap_and_config 2023-12-15T17:01:57.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.064+0000 7f403156c640 10 monclient: build_initial_monmap 2023-12-15T17:01:57.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.064+0000 7f403156c640 10 monclient: monmap: 2023-12-15T17:01:57.065 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:01:57.065 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:57.065 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:01:57.064906+0000 2023-12-15T17:01:57.065 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:01:57.064906+0000 2023-12-15T17:01:57.065 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:57.066 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:57.066 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:57.066 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:57.066 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:57.066 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:57.066 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:57.066 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 1 Processor -- start 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 1 -- start start 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 10 monclient: init 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 0x7f402c1427a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 10 monclient(hunting): picked mon.noname-b con 0x7f402c0b0ad0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f402c0728a0 addr v1:172.21.15.69:6789/0 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f402c070460 0x7f402c1493d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 10 monclient(hunting): picked mon.noname-a con 0x7f402c070460 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f402c070cf0 con 0x7f402c0728a0 2023-12-15T17:01:57.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f402c0b0eb0 con 0x7f402c070460 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f402c0ad340 con 0x7f402c0b0ad0 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.065+0000 7f403156c640 10 monclient(hunting): authenticate will time out at 1421.381094s 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.066+0000 7f402b7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f402c070460 0x7f402c1493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.066+0000 7f402affd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 0x7f402c1427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.066+0000 7f402affd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 0x7f402c1427a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36520/0 (socket says 172.21.15.69:36520) 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.066+0000 7f402affd640 1 -- 172.21.15.69:0/3484771998 learned_addr learned my addr 172.21.15.69:0/3484771998 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.066+0000 7f402affd640 10 monclient(hunting): get_auth_request con 0x7f402c0b0ad0 auth_method 0 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.066+0000 7f402affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.066+0000 7f402affd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.066+0000 7f402affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.067+0000 7f402b7fe640 10 monclient(hunting): get_auth_request con 0x7f402c070460 auth_method 0 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.067+0000 7f402b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:57.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.067+0000 7f402b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.067+0000 7f402b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.067+0000 7f402affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.067+0000 7f402affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.067+0000 7f402affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.067+0000 7f402b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.067+0000 7f402b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:57.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.067+0000 7f402b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.068+0000 7f402affd640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2023-12-15T17:01:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.069+0000 7f402affd640 1 -- 172.21.15.69:0/3484771998 >> v2:172.21.15.69:3300/0 conn(0x7f402c070460 msgr2=0x7f402c1493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.069+0000 7f402affd640 1 --2- 172.21.15.69:0/3484771998 >> v2:172.21.15.69:3300/0 conn(0x7f402c070460 0x7f402c1493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.069+0000 7f402affd640 1 -- 172.21.15.69:0/3484771998 >> v1:172.21.15.69:6789/0 conn(0x7f402c0728a0 legacy=0x7f402c06fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.069+0000 7f402affd640 10 monclient: _finish_hunting 0 2023-12-15T17:01:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.069+0000 7f402affd640 1 monclient: found mon.noname-b 2023-12-15T17:01:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.069+0000 7f402affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.069+0000 7f402affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:01:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.069+0000 7f402affd640 1 -- 172.21.15.69:0/3484771998 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402c0adcb0 con 0x7f402c0b0ad0 2023-12-15T17:01:57.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.070+0000 7f402affd640 1 --2- 172.21.15.69:0/3484771998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 0x7f402c1427a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f402000bb10 tx=0x7f402002e0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90373ebf75168154 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 1 -- 172.21.15.69:0/3484771998 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402002ee70 con 0x7f402c0b0ad0 2023-12-15T17:01:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 10 monclient: dump: 2023-12-15T17:01:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 10 monclient: _finish_auth 0 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 10 monclient: _check_auth_tickets 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 1 -- 172.21.15.69:0/3484771998 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4020036440 con 0x7f402c0b0ad0 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 1 -- 172.21.15.69:0/3484771998 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40200367d0 con 0x7f402c0b0ad0 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f4029ffb640 10 monclient: dump: 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:57.076 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:57.077 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f403156c640 5 monclient: authenticate success, global_id 4238 2023-12-15T17:01:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f403156c640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:57.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.074+0000 7f403156c640 4 set_mon_vals no callback set 2023-12-15T17:01:57.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 10 monclient: shutdown 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 1 -- 172.21.15.69:0/3484771998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 msgr2=0x7f402c1427a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 1 --2- 172.21.15.69:0/3484771998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 0x7f402c1427a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f402000bb10 tx=0x7f402002e0c0 comp rx=0 tx=0).stop 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 1 -- 172.21.15.69:0/3484771998 shutdown_connections 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 1 --2- 172.21.15.69:0/3484771998 >> v2:172.21.15.69:3300/0 conn(0x7f402c070460 0x7f402c1493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 1 --2- 172.21.15.69:0/3484771998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 0x7f402c1427a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 1 -- 172.21.15.69:0/3484771998 >> 172.21.15.69:0/3484771998 conn(0x7f402c0b3cb0 msgr2=0x7f402c0b0150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 1 -- 172.21.15.69:0/3484771998 shutdown_connections 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 1 -- 172.21.15.69:0/3484771998 wait complete. 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 10 monclient: build_initial_monmap 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.075+0000 7f403156c640 10 monclient: monmap: 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:01:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 librados: starting msgr at 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 librados: starting objecter 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 Processor -- start 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 -- start start 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 librados: setting wanted keys 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 librados: calling monclient init 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 10 monclient: init 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f402c070460 0x7f402c0b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 10 monclient(hunting): picked mon.noname-b con 0x7f402c070460 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f402c0728a0 0x7f402c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 10 monclient(hunting): picked mon.noname-a con 0x7f402c0728a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 0x7f402c1c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 10 monclient(hunting): picked mon.noname-c con 0x7f402c0b0ad0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402c0b2fa0 con 0x7f402c0728a0 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402c1c8b10 con 0x7f402c070460 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f402c0b12d0 con 0x7f402c0b0ad0 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.076+0000 7f403156c640 10 monclient(hunting): authenticate will time out at 1421.392336s 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.077+0000 7f402b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 0x7f402c1c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.077+0000 7f402a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f402c0728a0 0x7f402c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:57.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.077+0000 7f402affd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f402c070460 0x7f402c0b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.077+0000 7f402b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 0x7f402c1c91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36524/0 (socket says 172.21.15.69:36524) 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.077+0000 7f402b7fe640 1 -- 172.21.15.69:0/649825109 learned_addr learned my addr 172.21.15.69:0/649825109 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.077+0000 7f402b7fe640 10 monclient(hunting): get_auth_request con 0x7f402c0b0ad0 auth_method 0 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.077+0000 7f402b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.077+0000 7f402b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.077+0000 7f402b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402affd640 10 monclient(hunting): get_auth_request con 0x7f402c070460 auth_method 0 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402affd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402a7fc640 10 monclient(hunting): get_auth_request con 0x7f402c0728a0 auth_method 0 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:57.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.078+0000 7f402a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.081+0000 7f402affd640 10 monclient(hunting): handle_auth_done global_id 4267 payload 824 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.081+0000 7f402affd640 1 -- 172.21.15.69:0/649825109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 msgr2=0x7f402c1c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.081+0000 7f402affd640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 0x7f402c1c91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.081+0000 7f402affd640 1 -- 172.21.15.69:0/649825109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f402c0728a0 msgr2=0x7f402c0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.081+0000 7f402affd640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f402c0728a0 0x7f402c0b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.081+0000 7f402affd640 10 monclient: _finish_hunting 0 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.081+0000 7f402affd640 1 monclient: found mon.noname-b 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.081+0000 7f402affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.081+0000 7f402affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.081+0000 7f402affd640 1 -- 172.21.15.69:0/649825109 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402000b770 con 0x7f402c070460 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.083+0000 7f402affd640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f402c070460 0x7f402c0b28b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4020035000 tx=0x7f40200046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.083+0000 7f40137fe640 10 client.?.objecter ms_handle_connect 0x7f402c070460 2023-12-15T17:01:57.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.083+0000 7f40137fe640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:57.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f40137fe640 1 -- 172.21.15.69:0/649825109 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402003f910 con 0x7f402c070460 2023-12-15T17:01:57.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f40137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:57.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f40137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:57.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f40137fe640 10 monclient: dump: 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f40137fe640 10 monclient: _finish_auth 0 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f40137fe640 10 monclient: _check_auth_tickets 2023-12-15T17:01:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f40137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f40137fe640 1 -- 172.21.15.69:0/649825109 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402003fa70 con 0x7f402c070460 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f40137fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f40137fe640 1 -- 172.21.15.69:0/649825109 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402003fe20 con 0x7f402c070460 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f403156c640 5 monclient: authenticate success, global_id 4267 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f403156c640 10 monclient: _renew_subs 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f403156c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f403156c640 1 -- 172.21.15.69:0/649825109 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f402c142fe0 con 0x7f402c070460 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f40137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f40137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f40137fe640 10 monclient: dump: 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.086+0000 7f40297fa640 4 set_mon_vals no callback set 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f403156c640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f403156c640 10 monclient: _renew_subs 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f403156c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f403156c640 1 -- 172.21.15.69:0/649825109 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f402c1c96e0 con 0x7f402c070460 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f403156c640 1 librados: init done 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f403156c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f403156c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:57.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f403156c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:57.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.087+0000 7f403156c640 1 -- 172.21.15.69:0/649825109 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f402c0716d0 con 0x7f402c070460 2023-12-15T17:01:57.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.105+0000 7f40137fe640 1 -- 172.21.15.69:0/649825109 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f402003d070 con 0x7f402c070460 2023-12-15T17:01:57.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.105+0000 7f40137fe640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f401806cba0 0x7f401806f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:57.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.105+0000 7f402a7fc640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f401806cba0 0x7f401806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:57.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.105+0000 7f402a7fc640 10 monclient: get_auth_request con 0x7f401806cba0 auth_method 0 2023-12-15T17:01:57.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.106+0000 7f402a7fc640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f401806cba0 0x7f401806f060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f402c1c89d0 tx=0x7f401c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:57.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.106+0000 7f40137fe640 10 client.4267.objecter ms_handle_connect 0x7f401806cba0 2023-12-15T17:01:57.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.334+0000 7f40137fe640 1 -- 172.21.15.69:0/649825109 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f402007adc0 con 0x7f402c070460 2023-12-15T17:01:57.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.334+0000 7f40137fe640 10 client.4267.objecter ms_dispatch 0x7f402c000b90 osd_map(18..18 src has 1..18) v4 2023-12-15T17:01:57.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.334+0000 7f40137fe640 3 client.4267.objecter handle_osd_map got epochs [18,18] > 0 2023-12-15T17:01:57.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.334+0000 7f40137fe640 3 client.4267.objecter handle_osd_map decoding full epoch 18 2023-12-15T17:01:57.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.334+0000 7f40137fe640 20 client.4267.objecter dump_active .. 0 homeless 2023-12-15T17:01:57.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.344+0000 7f40137fe640 1 -- 172.21.15.69:0/649825109 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f4020077100 con 0x7f402c070460 2023-12-15T17:01:57.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.344+0000 7f40137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:57.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.344+0000 7f40137fe640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:57.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.526+0000 7f403156c640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-15T17:01:57.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.526+0000 7f403156c640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-15T17:01:57.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.527+0000 7f403156c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:01:57.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:57.527+0000 7f403156c640 1 -- 172.21.15.69:0/649825109 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f402c075b60 con 0x7f402c070460 2023-12-15T17:01:58.232 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.232+0000 7f40137fe640 1 -- 172.21.15.69:0/649825109 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v19) v1 ==== 109+0+0 (secure 0 0 0) 0x7f40200430a0 con 0x7f402c070460 2023-12-15T17:01:58.233 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.232+0000 7f40137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-15T17:01:58.233 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.232+0000 7f40137fe640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-12-15T17:01:58.233 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'rbd' created 2023-12-15T17:01:58.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 10 librados: watch_flush enter 2023-12-15T17:01:58.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 10 librados: watch_flush exit 2023-12-15T17:01:58.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 20 client.4267.objecter shutdown clearing up homeless session... 2023-12-15T17:01:58.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 10 client.4267.objecter successfully canceled tick 2023-12-15T17:01:58.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 -- 172.21.15.69:0/649825109 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f401806cba0 msgr2=0x7f401806f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:58.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f401806cba0 0x7f401806f060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f402c1c89d0 tx=0x7f401c009040 comp rx=0 tx=0).stop 2023-12-15T17:01:58.235 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 10 monclient: shutdown 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 -- 172.21.15.69:0/649825109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f402c070460 msgr2=0x7f402c0b28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f402c070460 0x7f402c0b28b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4020035000 tx=0x7f40200046c0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 -- 172.21.15.69:0/649825109 shutdown_connections 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f401806cba0 0x7f401806f060 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f402c0b0ad0 0x7f402c1c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f402c0728a0 0x7f402c0b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 --2- 172.21.15.69:0/649825109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f402c070460 0x7f402c0b28b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 -- 172.21.15.69:0/649825109 >> 172.21.15.69:0/649825109 conn(0x7f402c0b3cb0 msgr2=0x7f402c0af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 -- 172.21.15.69:0/649825109 shutdown_connections 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 -- 172.21.15.69:0/649825109 wait complete. 2023-12-15T17:01:58.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.234+0000 7f403156c640 1 librados: shutdown 2023-12-15T17:01:58.246 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-15T17:01:58.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.347+0000 7fa57b161640 10 monclient: get_monmap_and_config 2023-12-15T17:01:58.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.347+0000 7fa57b161640 10 monclient: build_initial_monmap 2023-12-15T17:01:58.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.347+0000 7fa57b161640 10 monclient: monmap: 2023-12-15T17:01:58.348 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:01:58.348 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:58.348 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:01:58.348080+0000 2023-12-15T17:01:58.349 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:01:58.348080+0000 2023-12-15T17:01:58.349 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:58.349 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:58.349 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:58.349 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:58.349 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:58.349 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:58.349 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:58.349 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:58.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.348+0000 7fa57b161640 1 Processor -- start 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.348+0000 7fa57b161640 1 -- start start 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.348+0000 7fa57b161640 10 monclient: init 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.348+0000 7fa57b161640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.348+0000 7fa57b161640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.348+0000 7fa57b161640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5740a8910 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.349+0000 7fa57b161640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5740a8cf0 addr v1:172.21.15.69:6789/0 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.349+0000 7fa57b161640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa57413f7a0 0x7fa5740b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.349+0000 7fa57b161640 10 monclient(hunting): picked mon.noname-c con 0x7fa57413f7a0 addr v2:172.21.15.192:3300/0 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.349+0000 7fa57b161640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5740b1ad0 con 0x7fa5740a8cf0 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.349+0000 7fa57b161640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5740b1db0 con 0x7fa5740a8910 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.349+0000 7fa57b161640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.349+0000 7fa57b161640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5740aa960 con 0x7fa57413f7a0 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.349+0000 7fa57b161640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:58.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.349+0000 7fa57b161640 10 monclient(hunting): authenticate will time out at 1422.664459s 2023-12-15T17:01:58.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.349+0000 7fa573fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa5740a8cf0 0x7fa57413f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33562/0 (socket says 172.21.15.69:33562) 2023-12-15T17:01:58.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.349+0000 7fa573fff640 1 -- 172.21.15.69:0/3586882559 learned_addr learned my addr 172.21.15.69:0/3586882559 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:01:58.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.350+0000 7fa5796fe640 1 --2- 172.21.15.69:0/3586882559 >> v2:172.21.15.192:3300/0 conn(0x7fa57413f7a0 0x7fa5740b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:58.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.351+0000 7fa5796fe640 10 monclient(hunting): get_auth_request con 0x7fa57413f7a0 auth_method 0 2023-12-15T17:01:58.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.351+0000 7fa5796fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:58.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.351+0000 7fa5796fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:58.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.351+0000 7fa5796fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:58.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.352+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946132155 0 0) 0x7fa5740b1ad0 con 0x7fa5740a8cf0 2023-12-15T17:01:58.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.352+0000 7fa5737fe640 10 monclient(hunting): _negotiate 2023-12-15T17:01:58.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.352+0000 7fa5737fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:58.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.352+0000 7fa5737fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:58.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.352+0000 7fa5737fe640 10 monclient(hunting): my global_id is 4251 2023-12-15T17:01:58.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.352+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa564003520 con 0x7fa5740a8cf0 2023-12-15T17:01:58.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.353+0000 7fa5796fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:58.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.353+0000 7fa5796fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:58.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.353+0000 7fa5796fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 110012989 0 0) 0x7fa564003520 con 0x7fa5740a8cf0 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 >> v2:172.21.15.192:3300/0 conn(0x7fa57413f7a0 msgr2=0x7fa5740b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 --2- 172.21.15.69:0/3586882559 >> v2:172.21.15.192:3300/0 conn(0x7fa57413f7a0 0x7fa5740b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 >> v1:172.21.15.192:6789/0 conn(0x7fa5740a8910 legacy=0x7fa574145ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient: _finish_hunting 0 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 monclient: found mon.noname-a-legacy 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.69:6789/0 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5741461d0 con 0x7fa5740a8cf0 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient: _finish_auth 0 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient: _check_auth_tickets 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:58.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa57b161640 5 monclient: authenticate success, global_id 4251 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa57b161640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2578675164 0 0) 0x7fa560001670 con 0x7fa5740a8cf0 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient: dump: 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] but i'm connected to v1:172.21.15.69:6789/0, reconnecting 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 >> v1:172.21.15.69:6789/0 conn(0x7fa5740a8cf0 legacy=0x7fa57413f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 --2- 172.21.15.69:0/3586882559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa564003a30 0x7fa564003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient(hunting): picked mon.a con 0x7fa564003a30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 --2- 172.21.15.69:0/3586882559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5640047c0 0x7fa564004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient(hunting): picked mon.c con 0x7fa5640047c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 --2- 172.21.15.69:0/3586882559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5640052d0 0x7fa56400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient(hunting): picked mon.b con 0x7fa5640052d0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740aa960 con 0x7fa564003a30 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56400c210 con 0x7fa5640052d0 2023-12-15T17:01:58.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:58.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56400c3d0 con 0x7fa5640047c0 2023-12-15T17:01:58.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:58.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa5737fe640 10 monclient(hunting): _finish_auth 0 2023-12-15T17:01:58.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.355+0000 7fa57b161640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:58.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.356+0000 7fa573fff640 1 --2- 172.21.15.69:0/3586882559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5640047c0 0x7fa564004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:58.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.356+0000 7fa573fff640 10 monclient(hunting): get_auth_request con 0x7fa5640047c0 auth_method 0 2023-12-15T17:01:58.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.356+0000 7fa573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:58.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.356+0000 7fa573fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:58.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.356+0000 7fa573fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:01:58.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.356+0000 7fa578efd640 1 --2- 172.21.15.69:0/3586882559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa564003a30 0x7fa564003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:58.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.357+0000 7fa573fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:58.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.357+0000 7fa573fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:58.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.357+0000 7fa573fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-15T17:01:58.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 10 monclient(hunting): handle_auth_done global_id 4251 payload 293 2023-12-15T17:01:58.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 1 -- 172.21.15.69:0/3586882559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5640052d0 msgr2=0x7fa56400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:58.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 1 --2- 172.21.15.69:0/3586882559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5640052d0 0x7fa56400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 1 -- 172.21.15.69:0/3586882559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa564003a30 msgr2=0x7fa564003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 1 --2- 172.21.15.69:0/3586882559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa564003a30 0x7fa564003e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 10 monclient: _finish_hunting 0 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 1 monclient: found mon.c 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 1 -- 172.21.15.69:0/3586882559 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa564003520 con 0x7fa5640047c0 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 10 monclient: _finish_auth 0 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 10 monclient: _check_auth_tickets 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa573fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.358+0000 7fa5796fe640 1 -- 172.21.15.69:0/3586882559 reap_dead start 2023-12-15T17:01:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.360+0000 7fa573fff640 1 --2- 172.21.15.69:0/3586882559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5640047c0 0x7fa564004c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa56000c0e0 tx=0x7fa5640057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7687f131d3ba1048 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:58.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560005ad0 con 0x7fa5640047c0 2023-12-15T17:01:58.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa5737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa5737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa5737fe640 10 monclient: dump: 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa560005c50 con 0x7fa5640047c0 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:01:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa5737fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa5737fe640 1 -- 172.21.15.69:0/3586882559 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56002c790 con 0x7fa5640047c0 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 4 set_mon_vals no callback set 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa5737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa5737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa5737fe640 10 monclient: dump: 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:58.366 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:58.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 10 monclient: shutdown 2023-12-15T17:01:58.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:58.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 1 -- 172.21.15.69:0/3586882559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5640047c0 msgr2=0x7fa564004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:58.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 1 --2- 172.21.15.69:0/3586882559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5640047c0 0x7fa564004c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa56000c0e0 tx=0x7fa5640057f0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 1 -- 172.21.15.69:0/3586882559 shutdown_connections 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 1 --2- 172.21.15.69:0/3586882559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5640047c0 0x7fa564004c40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 1 -- 172.21.15.69:0/3586882559 >> 172.21.15.69:0/3586882559 conn(0x7fa574061900 msgr2=0x7fa5740b76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 1 -- 172.21.15.69:0/3586882559 shutdown_connections 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.364+0000 7fa57b161640 1 -- 172.21.15.69:0/3586882559 wait complete. 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.365+0000 7fa57b161640 10 monclient: build_initial_monmap 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.365+0000 7fa57b161640 10 monclient: monmap: 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:58.370 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.365+0000 7fa57b161640 1 librados: starting msgr at 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.365+0000 7fa57b161640 1 librados: starting objecter 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.365+0000 7fa57b161640 1 Processor -- start 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.365+0000 7fa57b161640 1 -- start start 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.365+0000 7fa57b161640 1 librados: setting wanted keys 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.365+0000 7fa57b161640 1 librados: calling monclient init 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.365+0000 7fa57b161640 10 monclient: init 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5640047c0 0x7fa5741da640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 10 monclient(hunting): picked mon.noname-b con 0x7fa5640047c0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5740b03a0 0x7fa5741dab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 10 monclient(hunting): picked mon.noname-a con 0x7fa5740b03a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:58.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5740ae290 0x7fa5740ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 10 monclient(hunting): picked mon.noname-c con 0x7fa5740ae290 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5741da1b0 con 0x7fa5740b03a0 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740afca0 con 0x7fa5640047c0 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740acce0 con 0x7fa5740ae290 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.366+0000 7fa57b161640 10 monclient(hunting): authenticate will time out at 1422.682053s 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa573fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5740b03a0 0x7fa5741dab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa5796fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5740ae290 0x7fa5740ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa578efd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5640047c0 0x7fa5741da640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa573fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5740b03a0 0x7fa5741dab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42674/0 (socket says 172.21.15.69:42674) 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa573fff640 1 -- 172.21.15.69:0/767256684 learned_addr learned my addr 172.21.15.69:0/767256684 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa573fff640 10 monclient(hunting): get_auth_request con 0x7fa5740b03a0 auth_method 0 2023-12-15T17:01:58.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa573fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa573fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa5796fe640 10 monclient(hunting): get_auth_request con 0x7fa5740ae290 auth_method 0 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa5796fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa5796fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa5796fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa578efd640 10 monclient(hunting): get_auth_request con 0x7fa5640047c0 auth_method 0 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa578efd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa578efd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.367+0000 7fa578efd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.368+0000 7fa573fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.368+0000 7fa573fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.368+0000 7fa573fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.368+0000 7fa5796fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.368+0000 7fa5796fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:58.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.368+0000 7fa5796fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:58.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.368+0000 7fa578efd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:58.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.368+0000 7fa578efd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:58.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.368+0000 7fa578efd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:58.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.369+0000 7fa573fff640 10 monclient(hunting): handle_auth_done global_id 4254 payload 824 2023-12-15T17:01:58.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.370+0000 7fa573fff640 1 -- 172.21.15.69:0/767256684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5740ae290 msgr2=0x7fa5740ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:58.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.370+0000 7fa573fff640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5740ae290 0x7fa5740ae6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.370+0000 7fa573fff640 1 -- 172.21.15.69:0/767256684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5640047c0 msgr2=0x7fa5741da640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:58.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.370+0000 7fa573fff640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5640047c0 0x7fa5741da640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:58.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.370+0000 7fa573fff640 10 monclient: _finish_hunting 0 2023-12-15T17:01:58.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.370+0000 7fa573fff640 1 monclient: found mon.noname-a 2023-12-15T17:01:58.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.370+0000 7fa573fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:58.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.370+0000 7fa573fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:01:58.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.370+0000 7fa573fff640 1 -- 172.21.15.69:0/767256684 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5740aec20 con 0x7fa5740b03a0 2023-12-15T17:01:58.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.373+0000 7fa573fff640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5740b03a0 0x7fa5741dab80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa56002c400 tx=0x7fa5740b3bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:58.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.373+0000 7fa5717fa640 10 client.?.objecter ms_handle_connect 0x7fa5740b03a0 2023-12-15T17:01:58.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.373+0000 7fa5717fa640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 1 -- 172.21.15.69:0/767256684 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56002c790 con 0x7fa5740b03a0 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 10 monclient: dump: 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:58.381 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 10 monclient: _finish_auth 0 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 10 monclient: _check_auth_tickets 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 1 -- 172.21.15.69:0/767256684 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56003a070 con 0x7fa5740b03a0 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa57b161640 5 monclient: authenticate success, global_id 4254 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa57b161640 10 monclient: _renew_subs 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa57b161640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa57b161640 1 -- 172.21.15.69:0/767256684 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5741d6b20 con 0x7fa5740b03a0 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 1 -- 172.21.15.69:0/767256684 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56002caf0 con 0x7fa5740b03a0 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa5717fa640 10 monclient: dump: 2023-12-15T17:01:58.382 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.378+0000 7fa572ffd640 4 set_mon_vals no callback set 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.379+0000 7fa57b161640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.379+0000 7fa57b161640 10 monclient: _renew_subs 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.379+0000 7fa57b161640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.379+0000 7fa57b161640 1 -- 172.21.15.69:0/767256684 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5741d6fd0 con 0x7fa5740b03a0 2023-12-15T17:01:58.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.379+0000 7fa57b161640 1 librados: init done 2023-12-15T17:01:58.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.379+0000 7fa57b161640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:58.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.379+0000 7fa57b161640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:58.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.379+0000 7fa57b161640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:58.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.379+0000 7fa57b161640 1 -- 172.21.15.69:0/767256684 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5740aedb0 con 0x7fa5740b03a0 2023-12-15T17:01:58.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.622+0000 7fa5717fa640 1 -- 172.21.15.69:0/767256684 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa560032070 con 0x7fa5740b03a0 2023-12-15T17:01:58.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.622+0000 7fa5717fa640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55806cba0 0x7fa55806f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:58.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.622+0000 7fa578efd640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55806cba0 0x7fa55806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:58.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.622+0000 7fa578efd640 10 monclient: get_auth_request con 0x7fa55806cba0 auth_method 0 2023-12-15T17:01:58.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.623+0000 7fa578efd640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55806cba0 0x7fa55806f060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa5680017f0 tx=0x7fa568009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:58.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.623+0000 7fa5717fa640 10 client.4254.objecter ms_handle_connect 0x7fa55806cba0 2023-12-15T17:01:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.644+0000 7fa5717fa640 1 -- 172.21.15.69:0/767256684 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5199+0+0 (secure 0 0 0) 0x7fa56007a3c0 con 0x7fa5740b03a0 2023-12-15T17:01:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.644+0000 7fa5717fa640 10 client.4254.objecter ms_dispatch 0x7fa574000b90 osd_map(19..19 src has 1..19) v4 2023-12-15T17:01:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.644+0000 7fa5717fa640 3 client.4254.objecter handle_osd_map got epochs [19,19] > 0 2023-12-15T17:01:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.644+0000 7fa5717fa640 3 client.4254.objecter handle_osd_map decoding full epoch 19 2023-12-15T17:01:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.644+0000 7fa5717fa640 20 client.4254.objecter dump_active .. 0 homeless 2023-12-15T17:01:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.644+0000 7fa5717fa640 1 -- 172.21.15.69:0/767256684 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa560076000 con 0x7fa5740b03a0 2023-12-15T17:01:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.644+0000 7fa5717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.644+0000 7fa5717fa640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:58.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.820+0000 7fa57b161640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-15T17:01:58.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.820+0000 7fa57b161640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-15T17:01:58.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.820+0000 7fa57b161640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:58.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:58.820+0000 7fa57b161640 1 -- 172.21.15.69:0/767256684 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa5741da360 con 0x7fa5740b03a0 2023-12-15T17:01:59.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.190+0000 7fa5717fa640 1 -- 172.21.15.69:0/767256684 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v20) v1 ==== 173+0+0 (secure 0 0 0) 0x7fa5741da360 con 0x7fa5740b03a0 2023-12-15T17:01:59.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.190+0000 7fa5717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-15T17:01:59.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.190+0000 7fa5717fa640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-12-15T17:01:59.192 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-15T17:01:59.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.192+0000 7fa57b161640 10 librados: watch_flush enter 2023-12-15T17:01:59.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.192+0000 7fa57b161640 10 librados: watch_flush exit 2023-12-15T17:01:59.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.192+0000 7fa57b161640 20 client.4254.objecter shutdown clearing up homeless session... 2023-12-15T17:01:59.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.192+0000 7fa57b161640 10 client.4254.objecter successfully canceled tick 2023-12-15T17:01:59.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.192+0000 7fa57b161640 1 -- 172.21.15.69:0/767256684 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55806cba0 msgr2=0x7fa55806f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:59.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.192+0000 7fa57b161640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55806cba0 0x7fa55806f060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa5680017f0 tx=0x7fa568009040 comp rx=0 tx=0).stop 2023-12-15T17:01:59.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 10 monclient: shutdown 2023-12-15T17:01:59.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 1 -- 172.21.15.69:0/767256684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5740b03a0 msgr2=0x7fa5741dab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5740b03a0 0x7fa5741dab80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa56002c400 tx=0x7fa5740b3bc0 comp rx=0 tx=0).stop 2023-12-15T17:01:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 1 -- 172.21.15.69:0/767256684 shutdown_connections 2023-12-15T17:01:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55806cba0 0x7fa55806f060 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5740ae290 0x7fa5740ae6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5740b03a0 0x7fa5741dab80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 1 --2- 172.21.15.69:0/767256684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5640047c0 0x7fa5741da640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 1 -- 172.21.15.69:0/767256684 >> 172.21.15.69:0/767256684 conn(0x7fa574061900 msgr2=0x7fa57413d650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 1 -- 172.21.15.69:0/767256684 shutdown_connections 2023-12-15T17:01:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 1 -- 172.21.15.69:0/767256684 wait complete. 2023-12-15T17:01:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.193+0000 7fa57b161640 1 librados: shutdown 2023-12-15T17:01:59.206 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-15T17:01:59.206 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-15T17:01:59.207 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-15T17:01:59.207 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-15T17:01:59.207 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-15T17:01:59.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.317+0000 7f639a4e0640 10 monclient: get_monmap_and_config 2023-12-15T17:01:59.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.317+0000 7f639a4e0640 10 monclient: build_initial_monmap 2023-12-15T17:01:59.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.317+0000 7f639a4e0640 10 monclient: monmap: 2023-12-15T17:01:59.319 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:01:59.319 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:01:59.319148+0000 2023-12-15T17:01:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:01:59.319148+0000 2023-12-15T17:01:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:01:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:01:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:01:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:01:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:01:59.320 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:59.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.318+0000 7f639a4e0640 1 Processor -- start 2023-12-15T17:01:59.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.318+0000 7f639a4e0640 1 -- start start 2023-12-15T17:01:59.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.318+0000 7f639a4e0640 10 monclient: init 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63940a8910 addr v1:172.21.15.192:6789/0 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63940a9b70 0x7f639413f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 10 monclient(hunting): picked mon.noname-b con 0x7f63940a9b70 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6394141f90 0x7f63941423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 10 monclient(hunting): picked mon.noname-a con 0x7f6394141f90 addr v2:172.21.15.69:3300/0 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63940a8cf0 con 0x7f63940a8910 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63940aabd0 con 0x7f6394141f90 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63940aaeb0 con 0x7f63940a9b70 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:59.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f639a4e0640 10 monclient(hunting): authenticate will time out at 1423.635543s 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f63937fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63940a9b70 0x7f639413f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f6393fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f63940a8910 0x7f639413ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:60456/0 (socket says 172.21.15.69:60456) 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f6393fff640 1 -- 172.21.15.69:0/382129728 learned_addr learned my addr 172.21.15.69:0/382129728 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.319+0000 7f6398a7d640 1 --2- 172.21.15.69:0/382129728 >> v2:172.21.15.69:3300/0 conn(0x7f6394141f90 0x7f63941423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f63937fe640 10 monclient(hunting): get_auth_request con 0x7f63940a9b70 auth_method 0 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f63937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f63937fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f63937fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f6398a7d640 10 monclient(hunting): get_auth_request con 0x7f6394141f90 auth_method 0 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f6398a7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f6398a7d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f6398a7d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f63937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f63937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f63937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:59.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f6398a7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:59.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f6398a7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:59.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.320+0000 7f6398a7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:59.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.322+0000 7f63937fe640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2023-12-15T17:01:59.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.322+0000 7f63937fe640 1 -- 172.21.15.69:0/382129728 >> v2:172.21.15.69:3300/0 conn(0x7f6394141f90 msgr2=0x7f63941423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:01:59.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.322+0000 7f63937fe640 1 --2- 172.21.15.69:0/382129728 >> v2:172.21.15.69:3300/0 conn(0x7f6394141f90 0x7f63941423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:59.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.322+0000 7f63937fe640 1 -- 172.21.15.69:0/382129728 >> v1:172.21.15.192:6789/0 conn(0x7f63940a8910 legacy=0x7f639413ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:59.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.322+0000 7f63937fe640 10 monclient: _finish_hunting 0 2023-12-15T17:01:59.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.322+0000 7f63937fe640 1 monclient: found mon.noname-b 2023-12-15T17:01:59.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.322+0000 7f63937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:59.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.322+0000 7f63937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:01:59.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.322+0000 7f63937fe640 1 -- 172.21.15.69:0/382129728 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63940a9720 con 0x7f63940a9b70 2023-12-15T17:01:59.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.324+0000 7f63937fe640 1 --2- 172.21.15.69:0/382129728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63940a9b70 0x7f639413f370 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f63840097a0 tx=0x7f638402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a63ff22a476a35dc server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:59.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 1 -- 172.21.15.69:0/382129728 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384031830 con 0x7f63940a9b70 2023-12-15T17:01:59.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 10 monclient: dump: 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 10 monclient: _finish_auth 0 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:01:59.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 1 -- 172.21.15.69:0/382129728 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6384031990 con 0x7f63940a9b70 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 1 -- 172.21.15.69:0/382129728 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384031d20 con 0x7f63940a9b70 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f6392ffd640 10 monclient: dump: 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:59.331 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:59.332 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:59.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f639a4e0640 5 monclient: authenticate success, global_id 4247 2023-12-15T17:01:59.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f639a4e0640 10 monclient: get_monmap_and_config success 2023-12-15T17:01:59.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.327+0000 7f639a4e0640 4 set_mon_vals no callback set 2023-12-15T17:01:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.328+0000 7f639a4e0640 10 monclient: shutdown 2023-12-15T17:01:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.328+0000 7f639a4e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:01:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.328+0000 7f639a4e0640 1 -- 172.21.15.69:0/382129728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63940a9b70 msgr2=0x7f639413f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.328+0000 7f639a4e0640 1 --2- 172.21.15.69:0/382129728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63940a9b70 0x7f639413f370 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f63840097a0 tx=0x7f638402b8d0 comp rx=0 tx=0).stop 2023-12-15T17:01:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.328+0000 7f639a4e0640 1 -- 172.21.15.69:0/382129728 shutdown_connections 2023-12-15T17:01:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.328+0000 7f639a4e0640 1 --2- 172.21.15.69:0/382129728 >> v2:172.21.15.69:3300/0 conn(0x7f6394141f90 0x7f63941423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.328+0000 7f639a4e0640 1 --2- 172.21.15.69:0/382129728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63940a9b70 0x7f639413f370 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.328+0000 7f639a4e0640 1 -- 172.21.15.69:0/382129728 >> 172.21.15.69:0/382129728 conn(0x7f63940b7780 msgr2=0x7f63940b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.328+0000 7f639a4e0640 1 -- 172.21.15.69:0/382129728 shutdown_connections 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.328+0000 7f639a4e0640 1 -- 172.21.15.69:0/382129728 wait complete. 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.329+0000 7f639a4e0640 10 monclient: build_initial_monmap 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.329+0000 7f639a4e0640 10 monclient: monmap: 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:01:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.329+0000 7f639a4e0640 1 librados: starting msgr at 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.329+0000 7f639a4e0640 1 librados: starting objecter 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 1 Processor -- start 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 1 -- start start 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 1 librados: setting wanted keys 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 1 librados: calling monclient init 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 10 monclient: init 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 10 monclient: _reopen_session rank -1 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63940a8910 0x7f63941376a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 10 monclient(hunting): picked mon.noname-a con 0x7f63940a8910 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6394141f90 0x7f639413bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 10 monclient(hunting): picked mon.noname-b con 0x7f6394141f90 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:01:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394139110 0x7f639413fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 10 monclient(hunting): picked mon.noname-c con 0x7f6394139110 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639413ace0 con 0x7f63940a8910 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639413b8a0 con 0x7f6394141f90 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 10 monclient(hunting): start opening mon connection 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6394141c40 con 0x7f6394139110 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 10 monclient(hunting): _renew_subs 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.330+0000 7f639a4e0640 10 monclient(hunting): authenticate will time out at 1423.647189s 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6398a7d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394139110 0x7f639413fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6393fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63940a8910 0x7f63941376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f63937fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6394141f90 0x7f639413bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6398a7d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394139110 0x7f639413fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52756/0 (socket says 172.21.15.69:52756) 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6398a7d640 1 -- 172.21.15.69:0/3580426270 learned_addr learned my addr 172.21.15.69:0/3580426270 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:01:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6398a7d640 10 monclient(hunting): get_auth_request con 0x7f6394139110 auth_method 0 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6398a7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6398a7d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6398a7d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6393fff640 10 monclient(hunting): get_auth_request con 0x7f63940a8910 auth_method 0 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6393fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f6393fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f63937fe640 10 monclient(hunting): get_auth_request con 0x7f6394141f90 auth_method 0 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f63937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f63937fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.331+0000 7f63937fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.332+0000 7f6398a7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.332+0000 7f6398a7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:59.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.332+0000 7f6398a7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:59.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.332+0000 7f6393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:59.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.332+0000 7f6393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:59.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.332+0000 7f6393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:59.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.332+0000 7f63937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:01:59.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.332+0000 7f63937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:01:59.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.332+0000 7f63937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.334+0000 7f6393fff640 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.335+0000 7f6393fff640 1 -- 172.21.15.69:0/3580426270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394139110 msgr2=0x7f639413fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.335+0000 7f6393fff640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394139110 0x7f639413fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.335+0000 7f6393fff640 1 -- 172.21.15.69:0/3580426270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6394141f90 msgr2=0x7f639413bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.335+0000 7f6393fff640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6394141f90 0x7f639413bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.335+0000 7f6393fff640 10 monclient: _finish_hunting 0 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.335+0000 7f6393fff640 1 monclient: found mon.noname-a 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.335+0000 7f6393fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.335+0000 7f6393fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.335+0000 7f6393fff640 1 -- 172.21.15.69:0/3580426270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6384009400 con 0x7f63940a8910 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.337+0000 7f6393fff640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63940a8910 0x7f63941376a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f638800c470 tx=0x7f6388001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.337+0000 7f6390ff9640 10 client.?.objecter ms_handle_connect 0x7f63940a8910 2023-12-15T17:01:59.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.337+0000 7f6390ff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 1 -- 172.21.15.69:0/3580426270 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6388005ce0 con 0x7f63940a8910 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 10 monclient: dump: 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:59.344 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 10 monclient: _finish_auth 0 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 10 monclient: _check_auth_tickets 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 1 -- 172.21.15.69:0/3580426270 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6388005e40 con 0x7f63940a8910 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 1 -- 172.21.15.69:0/3580426270 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63880072d0 con 0x7f63940a8910 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f6390ff9640 10 monclient: dump: 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:01:59.345 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f639a4e0640 5 monclient: authenticate success, global_id 4260 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f639a4e0640 10 monclient: _renew_subs 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f639a4e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f639a4e0640 1 -- 172.21.15.69:0/3580426270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639413aec0 con 0x7f63940a8910 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f63927fc640 4 set_mon_vals no callback set 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f639a4e0640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f639a4e0640 10 monclient: _renew_subs 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f639a4e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f639a4e0640 1 -- 172.21.15.69:0/3580426270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639413a260 con 0x7f63940a8910 2023-12-15T17:01:59.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.342+0000 7f639a4e0640 1 librados: init done 2023-12-15T17:01:59.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.343+0000 7f639a4e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:59.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.343+0000 7f639a4e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:59.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.343+0000 7f639a4e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:59.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.343+0000 7f639a4e0640 1 -- 172.21.15.69:0/3580426270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6394075b60 con 0x7f63940a8910 2023-12-15T17:01:59.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.361+0000 7f6390ff9640 1 -- 172.21.15.69:0/3580426270 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6388007430 con 0x7f63940a8910 2023-12-15T17:01:59.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.361+0000 7f6390ff9640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f638006cb80 0x7f638006f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:01:59.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.362+0000 7f63937fe640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f638006cb80 0x7f638006f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:01:59.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.362+0000 7f63937fe640 10 monclient: get_auth_request con 0x7f638006cb80 auth_method 0 2023-12-15T17:01:59.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.362+0000 7f6390ff9640 10 client.4260.objecter ms_handle_connect 0x7f638006cb80 2023-12-15T17:01:59.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.362+0000 7f63937fe640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f638006cb80 0x7f638006f040 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6384036000 tx=0x7f638403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:01:59.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.365+0000 7f6390ff9640 1 -- 172.21.15.69:0/3580426270 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6388058160 con 0x7f63940a8910 2023-12-15T17:01:59.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.365+0000 7f6390ff9640 10 client.4260.objecter ms_dispatch 0x7f6394000b90 osd_map(20..20 src has 1..20) v4 2023-12-15T17:01:59.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.365+0000 7f6390ff9640 3 client.4260.objecter handle_osd_map got epochs [20,20] > 0 2023-12-15T17:01:59.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.365+0000 7f6390ff9640 3 client.4260.objecter handle_osd_map decoding full epoch 20 2023-12-15T17:01:59.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.366+0000 7f6390ff9640 20 client.4260.objecter dump_active .. 0 homeless 2023-12-15T17:01:59.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.656+0000 7f6390ff9640 1 -- 172.21.15.69:0/3580426270 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f63880544a0 con 0x7f63940a8910 2023-12-15T17:01:59.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.656+0000 7f6390ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:01:59.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.656+0000 7f6390ff9640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:01:59.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.915+0000 7f639a4e0640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-12-15T17:01:59.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.915+0000 7f639a4e0640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-12-15T17:01:59.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.915+0000 7f639a4e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:01:59.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:01:59.915+0000 7f639a4e0640 1 -- 172.21.15.69:0/3580426270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f639413a3f0 con 0x7f63940a8910 2023-12-15T17:01:59.938 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:09.883 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:01:59.994 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:09.939 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:00.027 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:09.972 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:00.038 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:09.977 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:00.079 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:10.024 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:00.099 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:10.038 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:00.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.234+0000 7f6390ff9640 1 -- 172.21.15.69:0/3580426270 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171167 (secure 0 0 0) 0x7f6388017090 con 0x7f63940a8910 2023-12-15T17:02:00.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.234+0000 7f6390ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-12-15T17:02:00.236 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.234+0000 7f6390ff9640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:00.239 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:02:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 10 librados: watch_flush enter 2023-12-15T17:02:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 10 librados: watch_flush exit 2023-12-15T17:02:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 20 client.4260.objecter shutdown clearing up homeless session... 2023-12-15T17:02:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 10 client.4260.objecter successfully canceled tick 2023-12-15T17:02:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 -- 172.21.15.69:0/3580426270 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f638006cb80 msgr2=0x7f638006f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f638006cb80 0x7f638006f040 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6384036000 tx=0x7f638403c040 comp rx=0 tx=0).stop 2023-12-15T17:02:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 10 monclient: shutdown 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 -- 172.21.15.69:0/3580426270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63940a8910 msgr2=0x7f63941376a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63940a8910 0x7f63941376a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f638800c470 tx=0x7f6388001e70 comp rx=0 tx=0).stop 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 -- 172.21.15.69:0/3580426270 shutdown_connections 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f638006cb80 0x7f638006f040 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394139110 0x7f639413fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6394141f90 0x7f639413bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 --2- 172.21.15.69:0/3580426270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63940a8910 0x7f63941376a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 -- 172.21.15.69:0/3580426270 >> 172.21.15.69:0/3580426270 conn(0x7f63940b7780 msgr2=0x7f639413f180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 -- 172.21.15.69:0/3580426270 shutdown_connections 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.237+0000 7f639a4e0640 1 -- 172.21.15.69:0/3580426270 wait complete. 2023-12-15T17:02:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.238+0000 7f639a4e0640 1 librados: shutdown 2023-12-15T17:02:00.254 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":5,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4171480121},{"type":"v1","addr":"172.21.15.192:6801","nonce":4171480121}]},"active_addr":"172.21.15.192:6801/4171480121","active_change":"2023-12-15T17:01:07.269381+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[{"gid":4111,"name":"y","mgr_features":4540701272860131327,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":3670782642}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":967341828}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":795461586}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":536294305}]}]} 2023-12-15T17:02:00.257 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-15T17:02:00.257 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-15T17:02:00.258 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-15T17:02:00.318 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:10.263 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:00.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.358+0000 7f66b3490640 10 monclient: get_monmap_and_config 2023-12-15T17:02:00.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.358+0000 7f66b3490640 10 monclient: build_initial_monmap 2023-12-15T17:02:00.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.358+0000 7f66b3490640 10 monclient: monmap: 2023-12-15T17:02:00.360 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:00.361 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:00.361 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:00.360316+0000 2023-12-15T17:02:00.361 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:00.360316+0000 2023-12-15T17:02:00.361 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:00.361 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:00.361 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:00.361 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:00.361 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:00.361 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:00.361 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:00.361 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.359+0000 7f66b3490640 1 Processor -- start 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.359+0000 7f66b3490640 1 -- start start 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.359+0000 7f66b3490640 10 monclient: init 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66ac0a8910 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f66ac0a9b70 0x7f66ac13f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 10 monclient(hunting): picked mon.noname-c con 0x7f66ac0a9b70 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66ac141f90 0x7f66ac1423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 10 monclient(hunting): picked mon.noname-a con 0x7f66ac141f90 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac0a8cf0 con 0x7f66ac0a8910 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:00.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ac0aabd0 con 0x7f66ac141f90 2023-12-15T17:02:00.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:00.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ac0aaeb0 con 0x7f66ac0a9b70 2023-12-15T17:02:00.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:00.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.360+0000 7f66b3490640 10 monclient(hunting): authenticate will time out at 1424.676697s 2023-12-15T17:02:00.367 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:10.307 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:00.414 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.412+0000 7f66b122c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f66ac0a8910 0x7f66ac13ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:60472/0 (socket says 172.21.15.69:60472) 2023-12-15T17:02:00.414 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.412+0000 7f66b122c640 1 -- 172.21.15.69:0/1868871004 learned_addr learned my addr 172.21.15.69:0/1868871004 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:00.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.423+0000 7f66b0a2b640 1 --2- 172.21.15.69:0/1868871004 >> v2:172.21.15.192:3300/0 conn(0x7f66ac0a9b70 0x7f66ac13f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:00.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.424+0000 7f66b0a2b640 10 monclient(hunting): get_auth_request con 0x7f66ac0a9b70 auth_method 0 2023-12-15T17:02:00.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.424+0000 7f66b0a2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:00.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.424+0000 7f66b0a2b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:00.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.424+0000 7f66b0a2b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:00.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.425+0000 7f66b0a2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:00.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.425+0000 7f66b0a2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:00.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.425+0000 7f66b0a2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:00.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.431+0000 7f66b0a2b640 10 monclient(hunting): handle_auth_done global_id 4285 payload 274 2023-12-15T17:02:00.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.431+0000 7f66b0a2b640 1 -- 172.21.15.69:0/1868871004 >> v2:172.21.15.69:3300/0 conn(0x7f66ac141f90 msgr2=0x7f66ac1423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:00.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.431+0000 7f66b0a2b640 1 --2- 172.21.15.69:0/1868871004 >> v2:172.21.15.69:3300/0 conn(0x7f66ac141f90 0x7f66ac1423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:00.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.431+0000 7f66b0a2b640 1 -- 172.21.15.69:0/1868871004 >> v1:172.21.15.192:6789/0 conn(0x7f66ac0a8910 legacy=0x7f66ac13ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:00.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.431+0000 7f66b0a2b640 10 monclient: _finish_hunting 0 2023-12-15T17:02:00.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.431+0000 7f66b0a2b640 1 monclient: found mon.noname-c 2023-12-15T17:02:00.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.431+0000 7f66b0a2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:00.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.431+0000 7f66b0a2b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:02:00.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.431+0000 7f66b0a2b640 1 -- 172.21.15.69:0/1868871004 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac0a9720 con 0x7f66ac0a9b70 2023-12-15T17:02:00.433 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:10.378 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:00.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.461+0000 7f66b0a2b640 1 --2- 172.21.15.69:0/1868871004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ac0a9b70 0x7f66ac13f370 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f66ac074590 tx=0x7f669c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b3b132599e265c2 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:00.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.476+0000 7f66a3fff640 1 -- 172.21.15.69:0/1868871004 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c031830 con 0x7f66ac0a9b70 2023-12-15T17:02:00.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.476+0000 7f66a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:00.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.476+0000 7f66a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.476+0000 7f66a3fff640 10 monclient: dump: 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66a3fff640 10 monclient: _finish_auth 0 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66a3fff640 10 monclient: _check_auth_tickets 2023-12-15T17:02:00.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66a3fff640 1 -- 172.21.15.69:0/1868871004 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c031990 con 0x7f66ac0a9b70 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66a3fff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66a3fff640 1 -- 172.21.15.69:0/1868871004 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c031d20 con 0x7f66ac0a9b70 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 5 monclient: authenticate success, global_id 4285 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 4 set_mon_vals no callback set 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66a3fff640 10 monclient: dump: 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:00.481 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:00.481 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:00.481 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:00.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 10 monclient: shutdown 2023-12-15T17:02:00.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:00.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 1 -- 172.21.15.69:0/1868871004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ac0a9b70 msgr2=0x7f66ac13f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:00.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 1 --2- 172.21.15.69:0/1868871004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ac0a9b70 0x7f66ac13f370 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f66ac074590 tx=0x7f669c02b8d0 comp rx=0 tx=0).stop 2023-12-15T17:02:00.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 1 -- 172.21.15.69:0/1868871004 shutdown_connections 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 1 --2- 172.21.15.69:0/1868871004 >> v2:172.21.15.69:3300/0 conn(0x7f66ac141f90 0x7f66ac1423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 1 --2- 172.21.15.69:0/1868871004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ac0a9b70 0x7f66ac13f370 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 1 -- 172.21.15.69:0/1868871004 >> 172.21.15.69:0/1868871004 conn(0x7f66ac0b7780 msgr2=0x7f66ac0b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 1 -- 172.21.15.69:0/1868871004 shutdown_connections 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.477+0000 7f66b3490640 1 -- 172.21.15.69:0/1868871004 wait complete. 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.478+0000 7f66b3490640 10 monclient: build_initial_monmap 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.478+0000 7f66b3490640 10 monclient: monmap: 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:00.485 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.478+0000 7f66b3490640 1 librados: starting msgr at 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.478+0000 7f66b3490640 1 librados: starting objecter 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.478+0000 7f66b3490640 1 Processor -- start 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.478+0000 7f66b3490640 1 -- start start 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.478+0000 7f66b3490640 1 librados: setting wanted keys 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.478+0000 7f66b3490640 1 librados: calling monclient init 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.478+0000 7f66b3490640 10 monclient: init 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66ac0a8910 0x7f66ac139950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 10 monclient(hunting): picked mon.noname-c con 0x7f66ac0a8910 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:00.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66ac141f90 0x7f66ac13b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 10 monclient(hunting): picked mon.noname-a con 0x7f66ac141f90 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ac139fc0 0x7f66ac137050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 10 monclient(hunting): picked mon.noname-b con 0x7f66ac139fc0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac13bb40 con 0x7f66ac141f90 2023-12-15T17:02:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac140d90 con 0x7f66ac139fc0 2023-12-15T17:02:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac139010 con 0x7f66ac0a8910 2023-12-15T17:02:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.479+0000 7f66b3490640 10 monclient(hunting): authenticate will time out at 1424.795920s 2023-12-15T17:02:00.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.494+0000 7f66b0a2b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66ac141f90 0x7f66ac13b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:00.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.498+0000 7f66b1a2d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ac139fc0 0x7f66ac137050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:00.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.504+0000 7f66b122c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66ac0a8910 0x7f66ac139950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:00.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.515+0000 7f66b122c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66ac0a8910 0x7f66ac139950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52766/0 (socket says 172.21.15.69:52766) 2023-12-15T17:02:00.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.516+0000 7f66b122c640 1 -- 172.21.15.69:0/3928043895 learned_addr learned my addr 172.21.15.69:0/3928043895 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:00.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.516+0000 7f66b122c640 10 monclient(hunting): get_auth_request con 0x7f66ac0a8910 auth_method 0 2023-12-15T17:02:00.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.516+0000 7f66b122c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:00.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.516+0000 7f66b122c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:00.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.516+0000 7f66b122c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:00.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.519+0000 7f66b1a2d640 10 monclient(hunting): get_auth_request con 0x7f66ac139fc0 auth_method 0 2023-12-15T17:02:00.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.519+0000 7f66b1a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:00.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.519+0000 7f66b1a2d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:00.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.519+0000 7f66b1a2d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:00.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.519+0000 7f66b122c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:00.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.519+0000 7f66b122c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:00.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.519+0000 7f66b122c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:00.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.525+0000 7f66b122c640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2023-12-15T17:02:00.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.525+0000 7f66b122c640 1 -- 172.21.15.69:0/3928043895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ac139fc0 msgr2=0x7f66ac137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:00.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.525+0000 7f66b122c640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ac139fc0 0x7f66ac137050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:00.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.525+0000 7f66b122c640 1 -- 172.21.15.69:0/3928043895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66ac141f90 msgr2=0x7f66ac13b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:00.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.525+0000 7f66b122c640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66ac141f90 0x7f66ac13b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:00.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.525+0000 7f66b122c640 10 monclient: _finish_hunting 0 2023-12-15T17:02:00.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.525+0000 7f66b122c640 1 monclient: found mon.noname-c 2023-12-15T17:02:00.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.525+0000 7f66b122c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:00.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.525+0000 7f66b122c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:02:00.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.525+0000 7f66b122c640 1 -- 172.21.15.69:0/3928043895 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c009400 con 0x7f66ac0a8910 2023-12-15T17:02:00.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.585+0000 7f66b122c640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66ac0a8910 0x7f66ac139950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f66a800c400 tx=0x7f66a800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:00.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.585+0000 7f66a1ffb640 10 client.?.objecter ms_handle_connect 0x7f66ac0a8910 2023-12-15T17:02:00.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.585+0000 7f66a1ffb640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:00.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 1 -- 172.21.15.69:0/3928043895 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8005ce0 con 0x7f66ac0a8910 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 10 monclient: dump: 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:00.603 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 10 monclient: _finish_auth 0 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 10 monclient: _check_auth_tickets 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 1 -- 172.21.15.69:0/3928043895 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a8005e40 con 0x7f66ac0a8910 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66b3490640 5 monclient: authenticate success, global_id 4253 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66b3490640 10 monclient: _renew_subs 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66b3490640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66b3490640 1 -- 172.21.15.69:0/3928043895 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac137590 con 0x7f66ac0a8910 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 1 -- 172.21.15.69:0/3928043895 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a80072d0 con 0x7f66ac0a8910 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:00.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a1ffb640 10 monclient: dump: 2023-12-15T17:02:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:00.605 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:00.606 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66a37fe640 4 set_mon_vals no callback set 2023-12-15T17:02:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66b3490640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.600+0000 7f66b3490640 10 monclient: _renew_subs 2023-12-15T17:02:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.601+0000 7f66b3490640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.601+0000 7f66b3490640 1 -- 172.21.15.69:0/3928043895 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac141b40 con 0x7f66ac0a8910 2023-12-15T17:02:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.601+0000 7f66b3490640 1 librados: init done 2023-12-15T17:02:00.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.602+0000 7f66b3490640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:00.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.602+0000 7f66b3490640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:00.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.602+0000 7f66b3490640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:00.607 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.602+0000 7f66b3490640 1 -- 172.21.15.69:0/3928043895 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ac13b310 con 0x7f66ac0a8910 2023-12-15T17:02:00.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.899+0000 7f66a1ffb640 1 -- 172.21.15.69:0/3928043895 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f66a80074f0 con 0x7f66ac0a8910 2023-12-15T17:02:00.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.899+0000 7f66a1ffb640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f669006cba0 0x7f669006f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:00.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.899+0000 7f66b0a2b640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f669006cba0 0x7f669006f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:00.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.899+0000 7f66b0a2b640 10 monclient: get_auth_request con 0x7f669006cba0 auth_method 0 2023-12-15T17:02:00.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.900+0000 7f66b0a2b640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f669006cba0 0x7f669006f060 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f669c036000 tx=0x7f669c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:00.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.900+0000 7f66a1ffb640 10 client.4253.objecter ms_handle_connect 0x7f669006cba0 2023-12-15T17:02:00.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.914+0000 7f66a1ffb640 1 -- 172.21.15.69:0/3928043895 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f66a80585d0 con 0x7f66ac0a8910 2023-12-15T17:02:00.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.914+0000 7f66a1ffb640 10 client.4253.objecter ms_dispatch 0x7f66ac000b90 osd_map(20..20 src has 1..20) v4 2023-12-15T17:02:00.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.914+0000 7f66a1ffb640 3 client.4253.objecter handle_osd_map got epochs [20,20] > 0 2023-12-15T17:02:00.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.914+0000 7f66a1ffb640 3 client.4253.objecter handle_osd_map decoding full epoch 20 2023-12-15T17:02:00.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.914+0000 7f66a1ffb640 20 client.4253.objecter dump_active .. 0 homeless 2023-12-15T17:02:00.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.914+0000 7f66a1ffb640 1 -- 172.21.15.69:0/3928043895 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f66a8054210 con 0x7f66ac0a8910 2023-12-15T17:02:00.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.914+0000 7f66a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:00.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:00.914+0000 7f66a1ffb640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:01.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.105+0000 7f66b3490640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:02:01.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.105+0000 7f66b3490640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:02:01.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.105+0000 7f66b3490640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:01.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.105+0000 7f66b3490640 1 -- 172.21.15.69:0/3928043895 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66ac075b60 con 0x7f66ac0a8910 2023-12-15T17:02:01.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.216+0000 7f66a1ffb640 1 -- 172.21.15.69:0/3928043895 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14664 (secure 0 0 0) 0x7f66a8017090 con 0x7f66ac0a8910 2023-12-15T17:02:01.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.216+0000 7f66a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:02:01.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.216+0000 7f66a1ffb640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:01.218 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:02:01.218 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":21,"fsid":"1a9f4b8c-183d-4435-93ef-5057f2b4aded","created":"2023-12-15T17:01:02.006221+0000","modified":"2023-12-15T17:02:00.207292+0000","last_up_change":"2023-12-15T17:01:50.252516+0000","last_in_change":"2023-12-15T17:01:08.340674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-15T17:01:49.711473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-15T17:01:57.539457+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51cce62a-d177-4e4b-a814-7e537407f397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6817","nonce":541068705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6819","nonce":541068705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6823","nonce":541068705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6821","nonce":541068705}]},"public_addr":"172.21.15.69:6817/541068705","cluster_addr":"172.21.15.69:6819/541068705","heartbeat_back_addr":"172.21.15.69:6823/541068705","heartbeat_front_addr":"172.21.15.69:6821/541068705","state":["exists","up"]},{"osd":1,"uuid":"1556db3a-b386-444f-8327-3dce2ad46d33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6809","nonce":1683202761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6811","nonce":1683202761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6815","nonce":1683202761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6813","nonce":1683202761}]},"public_addr":"172.21.15.69:6809/1683202761","cluster_addr":"172.21.15.69:6811/1683202761","heartbeat_back_addr":"172.21.15.69:6815/1683202761","heartbeat_front_addr":"172.21.15.69:6813/1683202761","state":["exists","up"]},{"osd":2,"uuid":"eaac40b5-9fc0-4e13-a322-763a9f3b685f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6801","nonce":3322029776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6803","nonce":3322029776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6807","nonce":3322029776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6805","nonce":3322029776}]},"public_addr":"172.21.15.69:6801/3322029776","cluster_addr":"172.21.15.69:6803/3322029776","heartbeat_back_addr":"172.21.15.69:6807/3322029776","heartbeat_front_addr":"172.21.15.69:6805/3322029776","state":["exists","up"]},{"osd":3,"uuid":"5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6825","nonce":882900488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6827","nonce":882900488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6831","nonce":882900488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6829","nonce":882900488}]},"public_addr":"172.21.15.69:6825/882900488","cluster_addr":"172.21.15.69:6827/882900488","heartbeat_back_addr":"172.21.15.69:6831/882900488","heartbeat_front_addr":"172.21.15.69:6829/882900488","state":["exists","up"]},{"osd":4,"uuid":"71221e7e-8713-4fc4-bef1-c1cbe40627da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6803","nonce":1420550374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6805","nonce":1420550374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6809","nonce":1420550374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6807","nonce":1420550374}]},"public_addr":"172.21.15.192:6803/1420550374","cluster_addr":"172.21.15.192:6805/1420550374","heartbeat_back_addr":"172.21.15.192:6809/1420550374","heartbeat_front_addr":"172.21.15.192:6807/1420550374","state":["exists","up"]},{"osd":5,"uuid":"2c9feecc-d3b5-429d-98fe-e6eddc91eb8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6811","nonce":2003653981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6813","nonce":2003653981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6817","nonce":2003653981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6815","nonce":2003653981}]},"public_addr":"172.21.15.192:6811/2003653981","cluster_addr":"172.21.15.192:6813/2003653981","heartbeat_back_addr":"172.21.15.192:6817/2003653981","heartbeat_front_addr":"172.21.15.192:6815/2003653981","state":["exists","up"]},{"osd":6,"uuid":"5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6819","nonce":2563835645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6821","nonce":2563835645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6825","nonce":2563835645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6823","nonce":2563835645}]},"public_addr":"172.21.15.192:6819/2563835645","cluster_addr":"172.21.15.192:6821/2563835645","heartbeat_back_addr":"172.21.15.192:6825/2563835645","heartbeat_front_addr":"172.21.15.192:6823/2563835645","state":["exists","up"]},{"osd":7,"uuid":"ac6e8a35-6df7-48d4-b59a-1f87eceaacfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6827","nonce":4283271495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6829","nonce":4283271495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6833","nonce":4283271495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6831","nonce":4283271495}]},"public_addr":"172.21.15.192:6827/4283271495","cluster_addr":"172.21.15.192:6829/4283271495","heartbeat_back_addr":"172.21.15.192:6833/4283271495","heartbeat_front_addr":"172.21.15.192:6831/4283271495","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.990860+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.734395+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:47.257292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:47.560872+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.054415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.592183+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.562550+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.670431+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T17:02:01.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 10 librados: watch_flush enter 2023-12-15T17:02:01.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 10 librados: watch_flush exit 2023-12-15T17:02:01.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 20 client.4253.objecter shutdown clearing up homeless session... 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 10 client.4253.objecter successfully canceled tick 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 -- 172.21.15.69:0/3928043895 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f669006cba0 msgr2=0x7f669006f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f669006cba0 0x7f669006f060 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f669c036000 tx=0x7f669c008030 comp rx=0 tx=0).stop 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 10 monclient: shutdown 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 -- 172.21.15.69:0/3928043895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66ac0a8910 msgr2=0x7f66ac139950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66ac0a8910 0x7f66ac139950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f66a800c400 tx=0x7f66a800c8c0 comp rx=0 tx=0).stop 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 -- 172.21.15.69:0/3928043895 shutdown_connections 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f669006cba0 0x7f669006f060 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ac139fc0 0x7f66ac137050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66ac141f90 0x7f66ac13b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 --2- 172.21.15.69:0/3928043895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66ac0a8910 0x7f66ac139950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 -- 172.21.15.69:0/3928043895 >> 172.21.15.69:0/3928043895 conn(0x7f66ac0b7780 msgr2=0x7f66ac1384c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 -- 172.21.15.69:0/3928043895 shutdown_connections 2023-12-15T17:02:01.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 -- 172.21.15.69:0/3928043895 wait complete. 2023-12-15T17:02:01.222 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.218+0000 7f66b3490640 1 librados: shutdown 2023-12-15T17:02:01.234 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-15T17:02:01.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-15T17:02:01.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.338+0000 7fe317821640 10 monclient: get_monmap_and_config 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.338+0000 7fe317821640 10 monclient: build_initial_monmap 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.338+0000 7fe317821640 10 monclient: monmap: 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:01.339903+0000 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:01.339903+0000 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:01.341 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:01.342 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:01.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.338+0000 7fe317821640 1 Processor -- start 2023-12-15T17:02:01.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 1 -- start start 2023-12-15T17:02:01.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 10 monclient: init 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe3100a8a30 0x7fe3101424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 10 monclient(hunting): picked mon.noname-a con 0x7fe3100a8a30 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3100a8e10 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 0x7fe310145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 10 monclient(hunting): picked mon.noname-b con 0x7fe310142a20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe310142eb0 con 0x7fe3100a8e10 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3100aaa10 con 0x7fe3100a8a30 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3100aac10 con 0x7fe310142a20 2023-12-15T17:02:01.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.339+0000 7fe317821640 10 monclient(hunting): authenticate will time out at 1425.656179s 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe315dbe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 0x7fe310145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe3155bd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe3100a8a30 0x7fe3101424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe3155bd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe3100a8a30 0x7fe3101424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58450/0 (socket says 172.21.15.69:58450) 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe3155bd640 1 -- 172.21.15.69:0/725996576 learned_addr learned my addr 172.21.15.69:0/725996576 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe315dbe640 10 monclient(hunting): get_auth_request con 0x7fe310142a20 auth_method 0 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe315dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe315dbe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe315dbe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe314dbc640 1 --1- 172.21.15.69:0/725996576 >> v1:172.21.15.192:6789/0 conn(0x7fe3100a8e10 0x7fe3101454a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:60486/0 (socket says 172.21.15.69:60486) 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe3155bd640 10 monclient(hunting): get_auth_request con 0x7fe3100a8a30 auth_method 0 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe3155bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe3155bd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.340+0000 7fe3155bd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:01.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.341+0000 7fe315dbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:01.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.341+0000 7fe315dbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:01.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.341+0000 7fe315dbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:01.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.341+0000 7fe3155bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:01.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.341+0000 7fe3155bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:01.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.341+0000 7fe3155bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:01.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.343+0000 7fe315dbe640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2023-12-15T17:02:01.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.343+0000 7fe315dbe640 1 -- 172.21.15.69:0/725996576 >> v2:172.21.15.69:3300/0 conn(0x7fe3100a8a30 msgr2=0x7fe3101424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:01.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.343+0000 7fe315dbe640 1 --2- 172.21.15.69:0/725996576 >> v2:172.21.15.69:3300/0 conn(0x7fe3100a8a30 0x7fe3101424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:01.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.343+0000 7fe315dbe640 1 -- 172.21.15.69:0/725996576 >> v1:172.21.15.192:6789/0 conn(0x7fe3100a8e10 legacy=0x7fe3101454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:01.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.343+0000 7fe315dbe640 10 monclient: _finish_hunting 0 2023-12-15T17:02:01.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.343+0000 7fe315dbe640 1 monclient: found mon.noname-b 2023-12-15T17:02:01.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.343+0000 7fe315dbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:01.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.343+0000 7fe315dbe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:01.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.343+0000 7fe315dbe640 1 -- 172.21.15.69:0/725996576 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3100aaef0 con 0x7fe310142a20 2023-12-15T17:02:01.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.345+0000 7fe315dbe640 1 --2- 172.21.15.69:0/725996576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 0x7fe310145bb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe308009d90 tx=0x7fe30802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bf7d5da28136179 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:01.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 1 -- 172.21.15.69:0/725996576 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30802cce0 con 0x7fe310142a20 2023-12-15T17:02:01.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:01.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:01.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 10 monclient: dump: 2023-12-15T17:02:01.352 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:01.352 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:01.352 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:01.352 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:01.352 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:01.352 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:01.352 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:01.352 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:01.353 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:01.353 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:01.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 10 monclient: _finish_auth 0 2023-12-15T17:02:01.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 10 monclient: _check_auth_tickets 2023-12-15T17:02:01.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:01.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 1 -- 172.21.15.69:0/725996576 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30802ce40 con 0x7fe310142a20 2023-12-15T17:02:01.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 5 monclient: authenticate success, global_id 4256 2023-12-15T17:02:01.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:01.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:01.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:01.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 4 set_mon_vals no callback set 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 1 -- 172.21.15.69:0/725996576 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308034660 con 0x7fe310142a20 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe307fff640 10 monclient: dump: 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:01.354 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:01.355 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:01.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 10 monclient: shutdown 2023-12-15T17:02:01.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:01.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 1 -- 172.21.15.69:0/725996576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 msgr2=0x7fe310145bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:01.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 1 --2- 172.21.15.69:0/725996576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 0x7fe310145bb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe308009d90 tx=0x7fe30802bf30 comp rx=0 tx=0).stop 2023-12-15T17:02:01.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 1 -- 172.21.15.69:0/725996576 shutdown_connections 2023-12-15T17:02:01.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 1 --2- 172.21.15.69:0/725996576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 0x7fe310145bb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:01.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 1 --2- 172.21.15.69:0/725996576 >> v2:172.21.15.69:3300/0 conn(0x7fe3100a8a30 0x7fe3101424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:01.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.349+0000 7fe317821640 1 -- 172.21.15.69:0/725996576 >> 172.21.15.69:0/725996576 conn(0x7fe310061a30 msgr2=0x7fe3100b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:01.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.350+0000 7fe317821640 1 -- 172.21.15.69:0/725996576 shutdown_connections 2023-12-15T17:02:01.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.350+0000 7fe317821640 1 -- 172.21.15.69:0/725996576 wait complete. 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.350+0000 7fe317821640 10 monclient: build_initial_monmap 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.350+0000 7fe317821640 10 monclient: monmap: 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:01.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.350+0000 7fe317821640 1 librados: starting msgr at 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 1 librados: starting objecter 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 1 Processor -- start 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 1 -- start start 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 1 librados: setting wanted keys 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 1 librados: calling monclient init 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 10 monclient: init 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3100a8a30 0x7fe3101d72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 10 monclient(hunting): picked mon.noname-b con 0x7fe3100a8a30 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3100a8e10 0x7fe3100af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 10 monclient(hunting): picked mon.noname-a con 0x7fe3100a8e10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 0x7fe3100afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:01.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 10 monclient(hunting): picked mon.noname-c con 0x7fe310142a20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3100aff70 con 0x7fe3100a8e10 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3100add70 con 0x7fe3100a8a30 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3100ae940 con 0x7fe310142a20 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.351+0000 7fe317821640 10 monclient(hunting): authenticate will time out at 1425.668214s 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe315dbe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 0x7fe3100afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe314dbc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3100a8e10 0x7fe3100af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe3155bd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3100a8a30 0x7fe3101d72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe315dbe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 0x7fe3100afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52780/0 (socket says 172.21.15.69:52780) 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe315dbe640 1 -- 172.21.15.69:0/2453492386 learned_addr learned my addr 172.21.15.69:0/2453492386 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:01.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe315dbe640 10 monclient(hunting): get_auth_request con 0x7fe310142a20 auth_method 0 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe315dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe315dbe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe315dbe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe314dbc640 10 monclient(hunting): get_auth_request con 0x7fe3100a8e10 auth_method 0 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe314dbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe314dbc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.352+0000 7fe314dbc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe3155bd640 10 monclient(hunting): get_auth_request con 0x7fe3100a8a30 auth_method 0 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe3155bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe3155bd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe3155bd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe315dbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe315dbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:01.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe315dbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:01.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe314dbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:01.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe314dbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:01.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe314dbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:01.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe3155bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:01.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe3155bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:01.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.353+0000 7fe3155bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.357+0000 7fe315dbe640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.357+0000 7fe315dbe640 1 -- 172.21.15.69:0/2453492386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3100a8a30 msgr2=0x7fe3101d72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.357+0000 7fe315dbe640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3100a8a30 0x7fe3101d72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.357+0000 7fe315dbe640 1 -- 172.21.15.69:0/2453492386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3100a8e10 msgr2=0x7fe3100af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.357+0000 7fe315dbe640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3100a8e10 0x7fe3100af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.357+0000 7fe315dbe640 10 monclient: _finish_hunting 0 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.357+0000 7fe315dbe640 1 monclient: found mon.noname-c 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.357+0000 7fe315dbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.357+0000 7fe315dbe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.357+0000 7fe315dbe640 1 -- 172.21.15.69:0/2453492386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe308009a70 con 0x7fe310142a20 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.361+0000 7fe315dbe640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 0x7fe3100afa30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe308009d90 tx=0x7fe308004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:01.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.361+0000 7fe305ffb640 10 client.?.objecter ms_handle_connect 0x7fe310142a20 2023-12-15T17:02:01.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.361+0000 7fe305ffb640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:01.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 1 -- 172.21.15.69:0/2453492386 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30803d600 con 0x7fe310142a20 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 10 monclient: dump: 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:01.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 10 monclient: _finish_auth 0 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 10 monclient: _check_auth_tickets 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 1 -- 172.21.15.69:0/2453492386 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30803d780 con 0x7fe310142a20 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 1 -- 172.21.15.69:0/2453492386 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30803db30 con 0x7fe310142a20 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe305ffb640 10 monclient: dump: 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:01.368 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe317821640 5 monclient: authenticate success, global_id 4259 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe317821640 10 monclient: _renew_subs 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe317821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe317821640 1 -- 172.21.15.69:0/2453492386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3100aefd0 con 0x7fe310142a20 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.364+0000 7fe3077fe640 4 set_mon_vals no callback set 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.365+0000 7fe317821640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.365+0000 7fe317821640 10 monclient: _renew_subs 2023-12-15T17:02:01.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.365+0000 7fe317821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:01.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.365+0000 7fe317821640 1 -- 172.21.15.69:0/2453492386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3100aebf0 con 0x7fe310142a20 2023-12-15T17:02:01.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.365+0000 7fe317821640 1 librados: init done 2023-12-15T17:02:01.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.366+0000 7fe317821640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:01.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.366+0000 7fe317821640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:01.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.366+0000 7fe317821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:01.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.366+0000 7fe317821640 1 -- 172.21.15.69:0/2453492386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe310067ac0 con 0x7fe310142a20 2023-12-15T17:02:01.623 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.621+0000 7fe305ffb640 1 -- 172.21.15.69:0/2453492386 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe30803b040 con 0x7fe310142a20 2023-12-15T17:02:01.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.622+0000 7fe305ffb640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe2f406cba0 0x7fe2f406f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:01.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.622+0000 7fe3155bd640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe2f406cba0 0x7fe2f406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:01.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.622+0000 7fe3155bd640 10 monclient: get_auth_request con 0x7fe2f406cba0 auth_method 0 2023-12-15T17:02:01.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.623+0000 7fe3155bd640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe2f406cba0 0x7fe2f406f060 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009bd0 tx=0x7fe30c009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:01.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.623+0000 7fe305ffb640 10 client.4259.objecter ms_handle_connect 0x7fe2f406cba0 2023-12-15T17:02:01.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.636+0000 7fe305ffb640 1 -- 172.21.15.69:0/2453492386 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe30807aeb0 con 0x7fe310142a20 2023-12-15T17:02:01.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.636+0000 7fe305ffb640 10 client.4259.objecter ms_dispatch 0x7fe310000b90 osd_map(21..21 src has 1..21) v4 2023-12-15T17:02:01.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.636+0000 7fe305ffb640 3 client.4259.objecter handle_osd_map got epochs [21,21] > 0 2023-12-15T17:02:01.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.636+0000 7fe305ffb640 3 client.4259.objecter handle_osd_map decoding full epoch 21 2023-12-15T17:02:01.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.636+0000 7fe305ffb640 20 client.4259.objecter dump_active .. 0 homeless 2023-12-15T17:02:01.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.636+0000 7fe305ffb640 1 -- 172.21.15.69:0/2453492386 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe3080771f0 con 0x7fe310142a20 2023-12-15T17:02:01.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.636+0000 7fe305ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:01.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.636+0000 7fe305ffb640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:01.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.834+0000 7fe317821640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:02:01.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.834+0000 7fe317821640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:02:01.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.834+0000 7fe317821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:01.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:01.834+0000 7fe317821640 1 -- 172.21.15.69:0/2453492386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3100aee10 con 0x7fe310142a20 2023-12-15T17:02:02.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.079+0000 7fe305ffb640 1 -- 172.21.15.69:0/2453492386 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14664 (secure 0 0 0) 0x7fe30804e090 con 0x7fe310142a20 2023-12-15T17:02:02.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.079+0000 7fe305ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:02:02.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.079+0000 7fe305ffb640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:02.082 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:02:02.082 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":21,"fsid":"1a9f4b8c-183d-4435-93ef-5057f2b4aded","created":"2023-12-15T17:01:02.006221+0000","modified":"2023-12-15T17:02:00.207292+0000","last_up_change":"2023-12-15T17:01:50.252516+0000","last_in_change":"2023-12-15T17:01:08.340674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-15T17:01:49.711473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-15T17:01:57.539457+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51cce62a-d177-4e4b-a814-7e537407f397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6817","nonce":541068705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6819","nonce":541068705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6823","nonce":541068705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6821","nonce":541068705}]},"public_addr":"172.21.15.69:6817/541068705","cluster_addr":"172.21.15.69:6819/541068705","heartbeat_back_addr":"172.21.15.69:6823/541068705","heartbeat_front_addr":"172.21.15.69:6821/541068705","state":["exists","up"]},{"osd":1,"uuid":"1556db3a-b386-444f-8327-3dce2ad46d33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6809","nonce":1683202761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6811","nonce":1683202761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6815","nonce":1683202761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6813","nonce":1683202761}]},"public_addr":"172.21.15.69:6809/1683202761","cluster_addr":"172.21.15.69:6811/1683202761","heartbeat_back_addr":"172.21.15.69:6815/1683202761","heartbeat_front_addr":"172.21.15.69:6813/1683202761","state":["exists","up"]},{"osd":2,"uuid":"eaac40b5-9fc0-4e13-a322-763a9f3b685f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6801","nonce":3322029776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6803","nonce":3322029776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6807","nonce":3322029776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6805","nonce":3322029776}]},"public_addr":"172.21.15.69:6801/3322029776","cluster_addr":"172.21.15.69:6803/3322029776","heartbeat_back_addr":"172.21.15.69:6807/3322029776","heartbeat_front_addr":"172.21.15.69:6805/3322029776","state":["exists","up"]},{"osd":3,"uuid":"5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6825","nonce":882900488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6827","nonce":882900488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6831","nonce":882900488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6829","nonce":882900488}]},"public_addr":"172.21.15.69:6825/882900488","cluster_addr":"172.21.15.69:6827/882900488","heartbeat_back_addr":"172.21.15.69:6831/882900488","heartbeat_front_addr":"172.21.15.69:6829/882900488","state":["exists","up"]},{"osd":4,"uuid":"71221e7e-8713-4fc4-bef1-c1cbe40627da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6803","nonce":1420550374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6805","nonce":1420550374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6809","nonce":1420550374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6807","nonce":1420550374}]},"public_addr":"172.21.15.192:6803/1420550374","cluster_addr":"172.21.15.192:6805/1420550374","heartbeat_back_addr":"172.21.15.192:6809/1420550374","heartbeat_front_addr":"172.21.15.192:6807/1420550374","state":["exists","up"]},{"osd":5,"uuid":"2c9feecc-d3b5-429d-98fe-e6eddc91eb8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6811","nonce":2003653981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6813","nonce":2003653981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6817","nonce":2003653981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6815","nonce":2003653981}]},"public_addr":"172.21.15.192:6811/2003653981","cluster_addr":"172.21.15.192:6813/2003653981","heartbeat_back_addr":"172.21.15.192:6817/2003653981","heartbeat_front_addr":"172.21.15.192:6815/2003653981","state":["exists","up"]},{"osd":6,"uuid":"5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6819","nonce":2563835645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6821","nonce":2563835645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6825","nonce":2563835645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6823","nonce":2563835645}]},"public_addr":"172.21.15.192:6819/2563835645","cluster_addr":"172.21.15.192:6821/2563835645","heartbeat_back_addr":"172.21.15.192:6825/2563835645","heartbeat_front_addr":"172.21.15.192:6823/2563835645","state":["exists","up"]},{"osd":7,"uuid":"ac6e8a35-6df7-48d4-b59a-1f87eceaacfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6827","nonce":4283271495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6829","nonce":4283271495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6833","nonce":4283271495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6831","nonce":4283271495}]},"public_addr":"172.21.15.192:6827/4283271495","cluster_addr":"172.21.15.192:6829/4283271495","heartbeat_back_addr":"172.21.15.192:6833/4283271495","heartbeat_front_addr":"172.21.15.192:6831/4283271495","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.990860+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.734395+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:47.257292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:47.560872+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.054415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.592183+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.562550+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.670431+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T17:02:02.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.081+0000 7fe317821640 10 librados: watch_flush enter 2023-12-15T17:02:02.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.081+0000 7fe317821640 10 librados: watch_flush exit 2023-12-15T17:02:02.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.081+0000 7fe317821640 20 client.4259.objecter shutdown clearing up homeless session... 2023-12-15T17:02:02.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.081+0000 7fe317821640 10 client.4259.objecter successfully canceled tick 2023-12-15T17:02:02.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.081+0000 7fe317821640 1 -- 172.21.15.69:0/2453492386 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe2f406cba0 msgr2=0x7fe2f406f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.081+0000 7fe317821640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe2f406cba0 0x7fe2f406f060 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009bd0 tx=0x7fe30c009240 comp rx=0 tx=0).stop 2023-12-15T17:02:02.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.081+0000 7fe317821640 10 monclient: shutdown 2023-12-15T17:02:02.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.081+0000 7fe317821640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.081+0000 7fe317821640 1 -- 172.21.15.69:0/2453492386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 msgr2=0x7fe3100afa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.081+0000 7fe317821640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 0x7fe3100afa30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe308009d90 tx=0x7fe308004a40 comp rx=0 tx=0).stop 2023-12-15T17:02:02.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.082+0000 7fe317821640 1 -- 172.21.15.69:0/2453492386 shutdown_connections 2023-12-15T17:02:02.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.082+0000 7fe317821640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe2f406cba0 0x7fe2f406f060 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.082+0000 7fe317821640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe310142a20 0x7fe3100afa30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.082+0000 7fe317821640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3100a8e10 0x7fe3100af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.082+0000 7fe317821640 1 --2- 172.21.15.69:0/2453492386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3100a8a30 0x7fe3101d72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.082+0000 7fe317821640 1 -- 172.21.15.69:0/2453492386 >> 172.21.15.69:0/2453492386 conn(0x7fe310061a30 msgr2=0x7fe3100b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.082+0000 7fe317821640 1 -- 172.21.15.69:0/2453492386 shutdown_connections 2023-12-15T17:02:02.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.082+0000 7fe317821640 1 -- 172.21.15.69:0/2453492386 wait complete. 2023-12-15T17:02:02.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.082+0000 7fe317821640 1 librados: shutdown 2023-12-15T17:02:02.096 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-15T17:02:02.096 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-15T17:02:02.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-15T17:02:02.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-15T17:02:02.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-15T17:02:02.098 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-15T17:02:02.098 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-15T17:02:02.099 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-15T17:02:02.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.291+0000 7f4cb1787640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.291+0000 7f4cb1787640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.294 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.291+0000 7f4cb1787640 10 monclient: monmap: 2023-12-15T17:02:02.294 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.294 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.294 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.292774+0000 2023-12-15T17:02:02.294 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.292774+0000 2023-12-15T17:02:02.294 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.294 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.294 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.295 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.295 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.295 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.295 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.295 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.294+0000 7f4cb1787640 1 Processor -- start 2023-12-15T17:02:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.294+0000 7f4cb1787640 1 -- start start 2023-12-15T17:02:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.294+0000 7f4cb1787640 10 monclient: init 2023-12-15T17:02:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-12-15T17:02:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4cac06f000 0x7f4cac06f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 10 monclient(hunting): picked mon.noname-b con 0x7f4cac06f000 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4cac06e380 0x7f4cac13af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 10 monclient(hunting): picked mon.noname-c con 0x7f4cac06e380 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4cac13b4c0 0x7f4cac0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 10 monclient(hunting): picked mon.noname-a con 0x7f4cac13b4c0 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cac070920 con 0x7f4cac13b4c0 2023-12-15T17:02:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cac06f980 con 0x7f4cac06e380 2023-12-15T17:02:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac06faf0 con 0x7f4cac06f000 2023-12-15T17:02:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.295+0000 7f4cb1787640 10 monclient(hunting): authenticate will time out at 1426.611591s 2023-12-15T17:02:02.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.296+0000 7f4cab7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4cac06e380 0x7f4cac13af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.296+0000 7f4cab7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4cac06e380 0x7f4cac13af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37128/0 (socket says 172.21.15.69:37128) 2023-12-15T17:02:02.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.296+0000 7f4cab7fe640 1 -- 172.21.15.69:0/2583984204 learned_addr learned my addr 172.21.15.69:0/2583984204 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.296+0000 7f4cab7fe640 10 monclient(hunting): get_auth_request con 0x7f4cac06e380 auth_method 0 2023-12-15T17:02:02.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.296+0000 7f4cab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.296+0000 7f4cab7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.296+0000 7f4cab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.297+0000 7f4cab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.297+0000 7f4cab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.297+0000 7f4cab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.300+0000 7f7158f4e640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.300+0000 7f7158f4e640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.300+0000 7f7158f4e640 10 monclient: monmap: 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.301940+0000 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.301940+0000 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.306 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.301+0000 7f7158f4e640 1 Processor -- start 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.301+0000 7f7158f4e640 1 -- start start 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.301+0000 7f7158f4e640 10 monclient: init 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71540b3f60 0x7f71540709d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 10 monclient(hunting): picked mon.noname-b con 0x7f71540b3f60 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7154070f10 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7154071350 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715408ad70 con 0x7f7154071350 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71540aee00 con 0x7f7154070f10 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715406ef50 con 0x7f71540b3f60 2023-12-15T17:02:02.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.302+0000 7f7158f4e640 10 monclient(hunting): authenticate will time out at 1426.618928s 2023-12-15T17:02:02.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f7152ffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f7154070f10 0x7f71540adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:60490/0 (socket says 172.21.15.69:60490) 2023-12-15T17:02:02.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f7152ffd640 1 -- 172.21.15.69:0/1855942730 learned_addr learned my addr 172.21.15.69:0/1855942730 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:02.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f4cab7fe640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2023-12-15T17:02:02.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f4cab7fe640 1 -- 172.21.15.69:0/2583984204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4cac06f000 msgr2=0x7f4cac06f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f4cab7fe640 1 --2- 172.21.15.69:0/2583984204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4cac06f000 0x7f4cac06f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f4cab7fe640 1 -- 172.21.15.69:0/2583984204 >> v2:172.21.15.69:3300/0 conn(0x7f4cac13b4c0 msgr2=0x7f4cac0710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f4cab7fe640 1 --2- 172.21.15.69:0/2583984204 >> v2:172.21.15.69:3300/0 conn(0x7f4cac13b4c0 0x7f4cac0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f4cab7fe640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f4cab7fe640 1 monclient: found mon.noname-c 2023-12-15T17:02:02.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f4cab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f4cab7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.303+0000 7f4cab7fe640 1 -- 172.21.15.69:0/2583984204 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac071610 con 0x7f4cac06e380 2023-12-15T17:02:02.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.305+0000 7f45a8505640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.305+0000 7f45a8505640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.305+0000 7f45a8505640 10 monclient: monmap: 2023-12-15T17:02:02.311 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.311 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.311 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.306897+0000 2023-12-15T17:02:02.311 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.306897+0000 2023-12-15T17:02:02.312 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.312 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.312 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.312 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.312 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.312 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.312 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.312 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.309+0000 7f4cab7fe640 1 --2- 172.21.15.69:0/2583984204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cac06e380 0x7f4cac13af80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4cac0b4310 tx=0x7f4ca002b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be2257fe88f7bcef server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.310+0000 7f45a8505640 1 Processor -- start 2023-12-15T17:02:02.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.310+0000 7f45a8505640 1 -- start start 2023-12-15T17:02:02.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.310+0000 7f45a8505640 10 monclient: init 2023-12-15T17:02:02.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-12-15T17:02:02.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45a00af360 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:02.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f45a013b800 0x7f45a0072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 10 monclient(hunting): picked mon.noname-a con 0x7f45a013b800 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:02.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f45a00adb40 0x7f45a006f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 10 monclient(hunting): picked mon.noname-c con 0x7f45a00adb40 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:02.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00726e0 con 0x7f45a00af360 2023-12-15T17:02:02.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a00af740 con 0x7f45a013b800 2023-12-15T17:02:02.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a0071690 con 0x7f45a00adb40 2023-12-15T17:02:02.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f45a8505640 10 monclient(hunting): authenticate will time out at 1426.627811s 2023-12-15T17:02:02.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.309+0000 7f71527fc640 1 -- 172.21.15.69:0/1855942730 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951526967 0 0) 0x7f71540aee00 con 0x7f7154070f10 2023-12-15T17:02:02.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.309+0000 7f71527fc640 10 monclient(hunting): _negotiate 2023-12-15T17:02:02.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.309+0000 7f71527fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.309+0000 7f71527fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.309+0000 7f71527fc640 10 monclient(hunting): my global_id is 4300 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.309+0000 7f71527fc640 1 -- 172.21.15.69:0/1855942730 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713c003520 con 0x7f7154070f10 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.310+0000 7f71537fe640 1 --2- 172.21.15.69:0/1855942730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71540b3f60 0x7f71540709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.310+0000 7f71537fe640 10 monclient(hunting): get_auth_request con 0x7f71540b3f60 auth_method 0 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.310+0000 7f71537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.310+0000 7f71537fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.310+0000 7f71537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f71537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f71537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.311+0000 7f71537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.312+0000 7f71527fc640 1 -- 172.21.15.69:0/1855942730 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601573613 0 0) 0x7f715408ad70 con 0x7f7154071350 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.312+0000 7f71527fc640 10 monclient(hunting): _negotiate 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.312+0000 7f71527fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.312+0000 7f71527fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.312+0000 7f71527fc640 10 monclient(hunting): my global_id is 4272 2023-12-15T17:02:02.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.312+0000 7f71527fc640 1 -- 172.21.15.69:0/1855942730 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71540aee00 con 0x7f7154071350 2023-12-15T17:02:02.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.314+0000 7f71537fe640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2023-12-15T17:02:02.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.314+0000 7f71537fe640 1 -- 172.21.15.69:0/1855942730 >> v1:172.21.15.192:6789/0 conn(0x7f7154070f10 legacy=0x7f71540adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.314+0000 7f71537fe640 1 -- 172.21.15.69:0/1855942730 >> v1:172.21.15.69:6789/0 conn(0x7f7154071350 legacy=0x7f71540ae6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.314+0000 7f71537fe640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.314+0000 7f71537fe640 1 monclient: found mon.noname-b 2023-12-15T17:02:02.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.314+0000 7f71537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.314+0000 7f71537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.314+0000 7f71537fe640 1 -- 172.21.15.69:0/1855942730 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715406fe80 con 0x7f71540b3f60 2023-12-15T17:02:02.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.314+0000 7f7e390aa640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.314+0000 7f7e390aa640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.314+0000 7f7e390aa640 10 monclient: monmap: 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.316255+0000 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.316255+0000 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.316+0000 7f7e390aa640 1 Processor -- start 2023-12-15T17:02:02.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.316+0000 7f7e390aa640 1 -- start start 2023-12-15T17:02:02.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.316+0000 7f7e390aa640 10 monclient: init 2023-12-15T17:02:02.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f7e390aa640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f7e390aa640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-12-15T17:02:02.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e2c034540 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:02.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e2c0c5640 0x7f7e2c0c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 10 monclient(hunting): picked mon.noname-c con 0x7f7e2c0c5640 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:02.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e2c0c60d0 0x7f7e2c0c6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 10 monclient(hunting): picked mon.noname-a con 0x7f7e2c0c60d0 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:02.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c0c5c40 con 0x7f7e2c034540 2023-12-15T17:02:02.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2c034920 con 0x7f7e2c0c60d0 2023-12-15T17:02:02.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2c02fd30 con 0x7f7e2c0c5640 2023-12-15T17:02:02.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f7e390aa640 10 monclient(hunting): authenticate will time out at 1426.634906s 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4caaffd640 1 -- 172.21.15.69:0/2583984204 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0031860 con 0x7f4cac06e380 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4caaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4caaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4caaffd640 10 monclient: dump: 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.322 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4caaffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4caaffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4caaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4cb1787640 5 monclient: authenticate success, global_id 4297 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4cb1787640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4caaffd640 1 -- 172.21.15.69:0/2583984204 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca00319c0 con 0x7f4cac06e380 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4caaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4cb1787640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.317+0000 7f4cb1787640 4 set_mon_vals no callback set 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f4cb1787640 10 monclient: shutdown 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f4cb1787640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f4cb1787640 1 -- 172.21.15.69:0/2583984204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cac06e380 msgr2=0x7f4cac13af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.318+0000 7f4cb1787640 1 --2- 172.21.15.69:0/2583984204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cac06e380 0x7f4cac13af80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4cac0b4310 tx=0x7f4ca002b900 comp rx=0 tx=0).stop 2023-12-15T17:02:02.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f4cb1787640 1 -- 172.21.15.69:0/2583984204 shutdown_connections 2023-12-15T17:02:02.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f4cb1787640 1 --2- 172.21.15.69:0/2583984204 >> v2:172.21.15.69:3300/0 conn(0x7f4cac13b4c0 0x7f4cac0710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f4cb1787640 1 --2- 172.21.15.69:0/2583984204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cac06e380 0x7f4cac13af80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f4cb1787640 1 --2- 172.21.15.69:0/2583984204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4cac06f000 0x7f4cac06f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f4cb1787640 1 -- 172.21.15.69:0/2583984204 >> 172.21.15.69:0/2583984204 conn(0x7f4cac06ace0 msgr2=0x7f4cac0b3570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.324 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f4cb1787640 1 -- 172.21.15.69:0/2583984204 shutdown_connections 2023-12-15T17:02:02.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7fdbc20ad640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7fdbc20ad640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.325 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7fdbc20ad640 10 monclient: monmap: 2023-12-15T17:02:02.325 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.325 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.325 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.321104+0000 2023-12-15T17:02:02.325 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.321104+0000 2023-12-15T17:02:02.325 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.325 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.325 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.325 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.326 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.326 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.326 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.326 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f4cb1787640 1 -- 172.21.15.69:0/2583984204 wait complete. 2023-12-15T17:02:02.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f71537fe640 1 --2- 172.21.15.69:0/1855942730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71540b3f60 0x7f71540709d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f71440097d0 tx=0x7f714402bb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c38a3c6b42e390fe server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f7e337fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e2c0c5640 0x7f7e2c0c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f7e337fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e2c0c5640 0x7f7e2c0c48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37148/0 (socket says 172.21.15.69:37148) 2023-12-15T17:02:02.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f7e337fe640 1 -- 172.21.15.69:0/36531574 learned_addr learned my addr 172.21.15.69:0/36531574 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f7e337fe640 10 monclient(hunting): get_auth_request con 0x7f7e2c0c5640 auth_method 0 2023-12-15T17:02:02.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f7e337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f7e337fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f7e337fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.327 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f4cb1787640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f4cb1787640 10 monclient: monmap: 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.328 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f4cb1787640 1 librados: starting msgr at 2023-12-15T17:02:02.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f7e388a9640 1 --2- 172.21.15.69:0/36531574 >> v2:172.21.15.69:3300/0 conn(0x7f7e2c0c60d0 0x7f7e2c0c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f7e388a9640 10 monclient(hunting): get_auth_request con 0x7f7e2c0c60d0 auth_method 0 2023-12-15T17:02:02.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f7e388a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f7e388a9640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f7e388a9640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f7e337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f7e337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.329 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f7e337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f4cb1787640 1 librados: starting objecter 2023-12-15T17:02:02.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7f4cb1787640 1 Processor -- start 2023-12-15T17:02:02.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f4cb1787640 1 -- start start 2023-12-15T17:02:02.330 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f4cb1787640 1 librados: setting wanted keys 2023-12-15T17:02:02.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f4cb1787640 1 librados: calling monclient init 2023-12-15T17:02:02.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f4cb1787640 10 monclient: init 2023-12-15T17:02:02.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f45a6aa2640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f45a00adb40 0x7f45a006f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f45a5aa0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f45a013b800 0x7f45a0072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.331 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f45a6aa2640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f45a00adb40 0x7f45a006f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37142/0 (socket says 172.21.15.69:37142) 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f45a6aa2640 1 -- 172.21.15.69:0/3273540366 learned_addr learned my addr 172.21.15.69:0/3273540366 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f45a6aa2640 10 monclient(hunting): get_auth_request con 0x7f45a00adb40 auth_method 0 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f45a6aa2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f45a6aa2640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.319+0000 7f45a6aa2640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f45a6aa2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f45a6aa2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f45a6aa2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f45a5aa0640 10 monclient(hunting): get_auth_request con 0x7f45a013b800 auth_method 0 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f45a5aa0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f45a5aa0640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7f45a5aa0640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:02:02.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cac06e380 0x7f4cac133750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 10 monclient(hunting): picked mon.noname-b con 0x7f4cac06e380 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4cac06f000 0x7f4cac138150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 10 monclient(hunting): picked mon.noname-c con 0x7f4cac06f000 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4cac13b4c0 0x7f4cac132710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 10 monclient(hunting): picked mon.noname-a con 0x7f4cac13b4c0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac13be70 con 0x7f4cac13b4c0 2023-12-15T17:02:02.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac1379e0 con 0x7f4cac06e380 2023-12-15T17:02:02.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac13b950 con 0x7f4cac06f000 2023-12-15T17:02:02.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.323+0000 7f7e337fe640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2023-12-15T17:02:02.335 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.323+0000 7f7e337fe640 1 -- 172.21.15.69:0/36531574 >> v2:172.21.15.69:3300/0 conn(0x7f7e2c0c60d0 msgr2=0x7f7e2c0c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7f4cb1787640 10 monclient(hunting): authenticate will time out at 1426.638871s 2023-12-15T17:02:02.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.323+0000 7f7e337fe640 1 --2- 172.21.15.69:0/36531574 >> v2:172.21.15.69:3300/0 conn(0x7f7e2c0c60d0 0x7f7e2c0c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.323+0000 7f7e337fe640 1 -- 172.21.15.69:0/36531574 >> v1:172.21.15.192:6789/0 conn(0x7f7e2c034540 legacy=0x7f7e2c0c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.323+0000 7f7e337fe640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.323+0000 7f7e337fe640 1 monclient: found mon.noname-c 2023-12-15T17:02:02.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.323+0000 7f7e337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.323+0000 7f7e337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.323+0000 7f7e337fe640 1 -- 172.21.15.69:0/36531574 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c030030 con 0x7f7e2c0c5640 2023-12-15T17:02:02.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7fdbc20ad640 1 Processor -- start 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7fdbc20ad640 1 -- start start 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.320+0000 7fdbc20ad640 10 monclient: init 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7fdbc20ad640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7fdbc20ad640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7fdbc20ad640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc0b0f20 0x7fdbbc0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7fdbc20ad640 10 monclient(hunting): picked mon.noname-b con 0x7fdbbc0b0f20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7fdbc20ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbbc1468b0 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7fdbc20ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbbc146cd0 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7fdbc20ad640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbbc06e870 con 0x7fdbbc1468b0 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7fdbc20ad640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbbc06eae0 con 0x7fdbbc146cd0 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7fdbc20ad640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7fdbc20ad640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0b3f10 con 0x7fdbbc0b0f20 2023-12-15T17:02:02.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.321+0000 7fdbc20ad640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.322+0000 7fdbc20ad640 10 monclient(hunting): authenticate will time out at 1426.638371s 2023-12-15T17:02:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.324+0000 7f45a5aa0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.324+0000 7f45a5aa0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.324+0000 7f45a5aa0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.324+0000 7fdbc10ab640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc0b0f20 0x7fdbbc0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.324+0000 7fdbc10ab640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc0b0f20 0x7fdbbc0b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52804/0 (socket says 172.21.15.69:52804) 2023-12-15T17:02:02.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.324+0000 7fdbc10ab640 1 -- 172.21.15.69:0/2111235687 learned_addr learned my addr 172.21.15.69:0/2111235687 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.324+0000 7fdbc10ab640 10 monclient(hunting): get_auth_request con 0x7fdbbc0b0f20 auth_method 0 2023-12-15T17:02:02.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.324+0000 7fdbc10ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.324+0000 7fdbc10ab640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.324+0000 7fdbc10ab640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f71527fc640 1 -- 172.21.15.69:0/1855942730 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714402c940 con 0x7f71540b3f60 2023-12-15T17:02:02.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f71527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f71527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f71527fc640 10 monclient: dump: 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.341 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f71527fc640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f71527fc640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f71527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f7158f4e640 5 monclient: authenticate success, global_id 4265 2023-12-15T17:02:02.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f7158f4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f71527fc640 1 -- 172.21.15.69:0/1855942730 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714402caa0 con 0x7f71540b3f60 2023-12-15T17:02:02.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f71527fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f7158f4e640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.326+0000 7f7158f4e640 4 set_mon_vals no callback set 2023-12-15T17:02:02.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7fdbc10ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7fdbc10ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7fdbc10ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f7158f4e640 10 monclient: shutdown 2023-12-15T17:02:02.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f7158f4e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f7158f4e640 1 -- 172.21.15.69:0/1855942730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71540b3f60 msgr2=0x7f71540709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f7158f4e640 1 --2- 172.21.15.69:0/1855942730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71540b3f60 0x7f71540709d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f71440097d0 tx=0x7f714402bb90 comp rx=0 tx=0).stop 2023-12-15T17:02:02.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f7158f4e640 1 -- 172.21.15.69:0/1855942730 shutdown_connections 2023-12-15T17:02:02.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f7158f4e640 1 --2- 172.21.15.69:0/1855942730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71540b3f60 0x7f71540709d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f7158f4e640 1 -- 172.21.15.69:0/1855942730 >> 172.21.15.69:0/1855942730 conn(0x7f715406ab30 msgr2=0x7f71540b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f45a6aa2640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2023-12-15T17:02:02.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f45a6aa2640 1 -- 172.21.15.69:0/3273540366 >> v2:172.21.15.69:3300/0 conn(0x7f45a013b800 msgr2=0x7f45a0072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f45a6aa2640 1 --2- 172.21.15.69:0/3273540366 >> v2:172.21.15.69:3300/0 conn(0x7f45a013b800 0x7f45a0072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f45a6aa2640 1 -- 172.21.15.69:0/3273540366 >> v1:172.21.15.192:6789/0 conn(0x7f45a00af360 legacy=0x7f45a013b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f45a6aa2640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f45a6aa2640 1 monclient: found mon.noname-c 2023-12-15T17:02:02.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f45a6aa2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f45a6aa2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.327+0000 7f45a6aa2640 1 -- 172.21.15.69:0/3273540366 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a006fcd0 con 0x7f45a00adb40 2023-12-15T17:02:02.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.328+0000 7f7158f4e640 1 -- 172.21.15.69:0/1855942730 shutdown_connections 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.328+0000 7f7158f4e640 1 -- 172.21.15.69:0/1855942730 wait complete. 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.329+0000 7f7158f4e640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.329+0000 7f7158f4e640 10 monclient: monmap: 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.346 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.347 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.329+0000 7f7158f4e640 1 librados: starting msgr at 2023-12-15T17:02:02.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.329+0000 7f7158f4e640 1 librados: starting objecter 2023-12-15T17:02:02.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.329+0000 7f7158f4e640 1 Processor -- start 2023-12-15T17:02:02.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.329+0000 7fdbc10ab640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2023-12-15T17:02:02.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.329+0000 7fdbc10ab640 1 -- 172.21.15.69:0/2111235687 >> v1:172.21.15.192:6789/0 conn(0x7fdbbc146cd0 legacy=0x7fdbbc149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.329+0000 7fdbc10ab640 1 -- 172.21.15.69:0/2111235687 >> v1:172.21.15.69:6789/0 conn(0x7fdbbc1468b0 legacy=0x7fdbbc1496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7fdbc10ab640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7fdbc10ab640 1 monclient: found mon.noname-b 2023-12-15T17:02:02.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7fdbc10ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7fdbc10ab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7fdbc10ab640 1 -- 172.21.15.69:0/2111235687 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc14a500 con 0x7fdbbc0b0f20 2023-12-15T17:02:02.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7f7158f4e640 1 -- start start 2023-12-15T17:02:02.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7f7158f4e640 1 librados: setting wanted keys 2023-12-15T17:02:02.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7f7158f4e640 1 librados: calling monclient init 2023-12-15T17:02:02.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7f7158f4e640 10 monclient: init 2023-12-15T17:02:02.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7f7158f4e640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7f7158f4e640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-15T17:02:02.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7f7158f4e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7154070f10 0x7f71541d0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.330+0000 7f7158f4e640 10 monclient(hunting): picked mon.noname-a con 0x7f7154070f10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.331+0000 7f7158f4e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7154071350 0x7f71541eb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.331+0000 7f4cab7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4cac06f000 0x7f4cac138150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.331+0000 7f7158f4e640 10 monclient(hunting): picked mon.noname-c con 0x7f7154071350 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.331+0000 7f7158f4e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71541cabd0 0x7f71541ebd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.331+0000 7f7158f4e640 10 monclient(hunting): picked mon.noname-b con 0x7f71541cabd0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.331+0000 7f7158f4e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.331+0000 7f7158f4e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71541d8c50 con 0x7f7154070f10 2023-12-15T17:02:02.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.331+0000 7f7158f4e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.331+0000 7f7158f4e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71541d8960 con 0x7f71541cabd0 2023-12-15T17:02:02.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f7158f4e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f7158f4e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71541cbe70 con 0x7f7154071350 2023-12-15T17:02:02.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f7158f4e640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f7158f4e640 10 monclient(hunting): authenticate will time out at 1426.648598s 2023-12-15T17:02:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f4cb0f86640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4cac13b4c0 0x7f4cac132710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f4cab7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4cac06f000 0x7f4cac138150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52812/0 (socket says 172.21.15.69:52812) 2023-12-15T17:02:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f4cab7fe640 1 -- 172.21.15.69:0/1189900783 learned_addr learned my addr 172.21.15.69:0/1189900783 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f4cab7fe640 10 monclient(hunting): get_auth_request con 0x7f4cac06f000 auth_method 0 2023-12-15T17:02:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f4cab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f4cab7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f4cab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f4cb0f86640 10 monclient(hunting): get_auth_request con 0x7f4cac13b4c0 auth_method 0 2023-12-15T17:02:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f4cb0f86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f4cb0f86640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.332+0000 7f4cb0f86640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.333+0000 7f4cab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.333+0000 7f4cab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.333+0000 7f4cab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.333+0000 7fb35e97c640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.333+0000 7fb35e97c640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.333+0000 7fb35e97c640 10 monclient: monmap: 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.334874+0000 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.334874+0000 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.355 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.334+0000 7fb35e97c640 1 Processor -- start 2023-12-15T17:02:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.334+0000 7fb35e97c640 1 -- start start 2023-12-15T17:02:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.334+0000 7fb35e97c640 10 monclient: init 2023-12-15T17:02:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.334+0000 7f80fb59e640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.334+0000 7f80fb59e640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.334+0000 7f80fb59e640 10 monclient: monmap: 2023-12-15T17:02:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.336101+0000 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.336101+0000 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.334+0000 7f4cb0f86640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.334+0000 7f4cb0f86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.334+0000 7f4cb0f86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7f80fb59e640 1 Processor -- start 2023-12-15T17:02:02.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7f80fb59e640 1 -- start start 2023-12-15T17:02:02.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7f80fb59e640 10 monclient: init 2023-12-15T17:02:02.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-12-15T17:02:02.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb3580a8a30 0x7fb3581424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 10 monclient(hunting): picked mon.noname-c con 0x7fb3580a8a30 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:02.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3580a8e10 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:02.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb358142a20 0x7fb358145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 10 monclient(hunting): picked mon.noname-b con 0x7fb358142a20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb358142eb0 con 0x7fb3580a8e10 2023-12-15T17:02:02.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3580aaa10 con 0x7fb3580a8a30 2023-12-15T17:02:02.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580aac10 con 0x7fb358142a20 2023-12-15T17:02:02.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7fb35e97c640 10 monclient(hunting): authenticate will time out at 1426.652305s 2023-12-15T17:02:02.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.335+0000 7f71537fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7154070f10 0x7f71541d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-12-15T17:02:02.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f4038d80 0x7f80f40c5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f80f4038d80 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f80f4039160 0x7f80f40c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f80f4039160 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f80f40c5770 0x7f80f40c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f80f40c5770 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f4037580 con 0x7f80f40c5770 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f40c7e70 con 0x7f80f4039160 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f4035870 con 0x7f80f4038d80 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f80fb59e640 10 monclient(hunting): authenticate will time out at 1426.652956s 2023-12-15T17:02:02.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f71537fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7154070f10 0x7f71541d0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58502/0 (socket says 172.21.15.69:58502) 2023-12-15T17:02:02.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f71537fe640 1 -- 172.21.15.69:0/2808440034 learned_addr learned my addr 172.21.15.69:0/2808440034 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f71537fe640 10 monclient(hunting): get_auth_request con 0x7f7154070f10 auth_method 0 2023-12-15T17:02:02.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f71537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f71537fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.336+0000 7f71537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.337+0000 7f45a6aa2640 1 --2- 172.21.15.69:0/3273540366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45a00adb40 0x7f45a006f790 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45a00adad0 tx=0x7f45980303d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be0f10f5260b579c server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.337+0000 7f7e337fe640 1 --2- 172.21.15.69:0/36531574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e2c0c5640 0x7f7e2c0c48c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0c71c0 tx=0x7f7e2002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b387ddd06bd38ed server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.338+0000 7f7153fff640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71541cabd0 0x7f71541ebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.338+0000 7f7153fff640 10 monclient(hunting): get_auth_request con 0x7f71541cabd0 auth_method 0 2023-12-15T17:02:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.338+0000 7f7153fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.338+0000 7f7153fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.338+0000 7f7153fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.339+0000 7f4cb0f86640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2023-12-15T17:02:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.339+0000 7f4cb0f86640 1 -- 172.21.15.69:0/1189900783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4cac06f000 msgr2=0x7f4cac138150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.339+0000 7f4cb0f86640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4cac06f000 0x7f4cac138150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.339+0000 7f4cb0f86640 1 -- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cac06e380 msgr2=0x7f4cac133750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.339+0000 7f4cb0f86640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cac06e380 0x7f4cac133750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.339+0000 7f4cb0f86640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.339+0000 7f4cb0f86640 1 monclient: found mon.noname-a 2023-12-15T17:02:02.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.339+0000 7f4cb0f86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.340+0000 7f4cb0f86640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.340+0000 7f4cb0f86640 1 -- 172.21.15.69:0/1189900783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca00094d0 con 0x7f4cac13b4c0 2023-12-15T17:02:02.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.341+0000 7f7153fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.341+0000 7f7153fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.341+0000 7f7153fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.342+0000 7fdbc10ab640 1 --2- 172.21.15.69:0/2111235687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc0b0f20 0x7fdbbc0b2110 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80098a0 tx=0x7fdbb802b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81543b2b7ca2ad47 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f71537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f71537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f71537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 1 -- 172.21.15.69:0/36531574 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e200317e0 con 0x7f7e2c0c5640 2023-12-15T17:02:02.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.369 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 10 monclient: dump: 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 1 -- 172.21.15.69:0/36531574 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e20031940 con 0x7f7e2c0c5640 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 1 -- 172.21.15.69:0/36531574 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20031cd0 con 0x7f7e2c0c5640 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e32ffd640 10 monclient: dump: 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.371 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.372 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e390aa640 5 monclient: authenticate success, global_id 4306 2023-12-15T17:02:02.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e390aa640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f7e390aa640 4 set_mon_vals no callback set 2023-12-15T17:02:02.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.344+0000 7f80f9d9b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f80f4039160 0x7f80f40c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f7e390aa640 10 monclient: shutdown 2023-12-15T17:02:02.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f7e390aa640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f7e390aa640 1 -- 172.21.15.69:0/36531574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e2c0c5640 msgr2=0x7f7e2c0c48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f7e390aa640 1 --2- 172.21.15.69:0/36531574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e2c0c5640 0x7f7e2c0c48c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0c71c0 tx=0x7f7e2002b880 comp rx=0 tx=0).stop 2023-12-15T17:02:02.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f80f9d9b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f80f4039160 0x7f80f40c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37192/0 (socket says 172.21.15.69:37192) 2023-12-15T17:02:02.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f80f9d9b640 1 -- 172.21.15.69:0/1918372231 learned_addr learned my addr 172.21.15.69:0/1918372231 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f80f9d9b640 10 monclient(hunting): get_auth_request con 0x7f80f4039160 auth_method 0 2023-12-15T17:02:02.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f80f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f80f9d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f80f9d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f7e390aa640 1 -- 172.21.15.69:0/36531574 shutdown_connections 2023-12-15T17:02:02.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f7e390aa640 1 --2- 172.21.15.69:0/36531574 >> v2:172.21.15.69:3300/0 conn(0x7f7e2c0c60d0 0x7f7e2c0c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f7e390aa640 1 --2- 172.21.15.69:0/36531574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e2c0c5640 0x7f7e2c0c48c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f7e390aa640 1 -- 172.21.15.69:0/36531574 >> 172.21.15.69:0/36531574 conn(0x7f7e2c036610 msgr2=0x7f7e2c005eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f7e390aa640 1 -- 172.21.15.69:0/36531574 shutdown_connections 2023-12-15T17:02:02.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.345+0000 7f7e390aa640 1 -- 172.21.15.69:0/36531574 wait complete. 2023-12-15T17:02:02.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.346+0000 7f7e390aa640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.346+0000 7f7e390aa640 10 monclient: monmap: 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7e390aa640 1 librados: starting msgr at 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7e390aa640 1 librados: starting objecter 2023-12-15T17:02:02.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7e390aa640 1 Processor -- start 2023-12-15T17:02:02.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7153fff640 10 monclient(hunting): handle_auth_done global_id 4312 payload 824 2023-12-15T17:02:02.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7153fff640 1 -- 172.21.15.69:0/2808440034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7154071350 msgr2=0x7f71541eb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7153fff640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7154071350 0x7f71541eb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7153fff640 1 -- 172.21.15.69:0/2808440034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7154070f10 msgr2=0x7f71541d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7153fff640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7154070f10 0x7f71541d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7153fff640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7153fff640 1 monclient: found mon.noname-b 2023-12-15T17:02:02.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7153fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7153fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7153fff640 1 -- 172.21.15.69:0/2808440034 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7144009400 con 0x7f71541cabd0 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7e390aa640 1 -- start start 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7e390aa640 1 librados: setting wanted keys 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7e390aa640 1 librados: calling monclient init 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.347+0000 7f7e390aa640 10 monclient: init 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2c034540 0x7f7e2c02ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 10 monclient(hunting): picked mon.noname-a con 0x7f7e2c034540 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c0c5640 0x7f7e2c15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 10 monclient(hunting): picked mon.noname-c con 0x7f7e2c0c5640 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e2c0c60d0 0x7f7e2c029100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 10 monclient(hunting): picked mon.noname-b con 0x7f7e2c0c60d0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c14ef90 con 0x7f7e2c034540 2023-12-15T17:02:02.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c02e8c0 con 0x7f7e2c0c60d0 2023-12-15T17:02:02.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c15c470 con 0x7f7e2c0c5640 2023-12-15T17:02:02.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f7e390aa640 10 monclient(hunting): authenticate will time out at 1426.664922s 2023-12-15T17:02:02.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.348+0000 7f80fa59c640 1 --2- 172.21.15.69:0/1918372231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f4038d80 0x7f80f40c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.349+0000 7f80f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.349+0000 7f80f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.349+0000 7f80f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.350+0000 7fb35cf19640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb358142a20 0x7fb358145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.350+0000 7f80fa59c640 10 monclient(hunting): get_auth_request con 0x7f80f4038d80 auth_method 0 2023-12-15T17:02:02.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.350+0000 7f80fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.350+0000 7f80fa59c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.350+0000 7f80fa59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.351+0000 7f7153fff640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71541cabd0 0x7f71541ebd50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f714800c3f0 tx=0x7f7148001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.351+0000 7f713bfff640 10 client.?.objecter ms_handle_connect 0x7f71541cabd0 2023-12-15T17:02:02.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.351+0000 7f713bfff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:02.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.353+0000 7f80fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.353+0000 7f80fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.353+0000 7f80fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.353+0000 7f4cb0f86640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4cac13b4c0 0x7f4cac132710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400be20 tx=0x7f4ca400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.353+0000 7f4ca8ff9640 10 client.?.objecter ms_handle_connect 0x7f4cac13b4c0 2023-12-15T17:02:02.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.353+0000 7f4ca8ff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:02.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.354+0000 7fb35cf19640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb358142a20 0x7fb358145bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52830/0 (socket says 172.21.15.69:52830) 2023-12-15T17:02:02.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.354+0000 7fb35cf19640 1 -- 172.21.15.69:0/1443912458 learned_addr learned my addr 172.21.15.69:0/1443912458 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.354+0000 7fb35cf19640 10 monclient(hunting): get_auth_request con 0x7fb358142a20 auth_method 0 2023-12-15T17:02:02.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.354+0000 7fb35cf19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.354+0000 7fb35cf19640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.354+0000 7fb35cf19640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.354+0000 7fb35cf19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.354+0000 7fb35cf19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.354+0000 7fb35cf19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.354+0000 7f7e337fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c0c5640 0x7f7e2c15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.355+0000 7f80fad9d640 1 --2- 172.21.15.69:0/1918372231 >> v2:172.21.15.69:3300/0 conn(0x7f80f40c5770 0x7f80f40c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.356+0000 7f80fa59c640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2023-12-15T17:02:02.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.356+0000 7f80fa59c640 1 -- 172.21.15.69:0/1918372231 >> v2:172.21.15.192:3300/0 conn(0x7f80f4039160 msgr2=0x7f80f40c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.356+0000 7f80fa59c640 1 --2- 172.21.15.69:0/1918372231 >> v2:172.21.15.192:3300/0 conn(0x7f80f4039160 0x7f80f40c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.356+0000 7f80fa59c640 1 -- 172.21.15.69:0/1918372231 >> v2:172.21.15.69:3300/0 conn(0x7f80f40c5770 msgr2=0x7f80f40c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.356+0000 7f80fa59c640 1 --2- 172.21.15.69:0/1918372231 >> v2:172.21.15.69:3300/0 conn(0x7f80f40c5770 0x7f80f40c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.356+0000 7f80fa59c640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.356+0000 7f80fa59c640 1 monclient: found mon.noname-b 2023-12-15T17:02:02.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.356+0000 7f80fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.356+0000 7f80fa59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.356+0000 7f80fa59c640 1 -- 172.21.15.69:0/1918372231 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f4036070 con 0x7f80f4038d80 2023-12-15T17:02:02.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.357+0000 7f5ef4c09640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.357+0000 7f5ef4c09640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.357+0000 7f7e337fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c0c5640 0x7f7e2c15bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52838/0 (socket says 172.21.15.69:52838) 2023-12-15T17:02:02.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.357+0000 7f7e337fe640 1 -- 172.21.15.69:0/2644306268 learned_addr learned my addr 172.21.15.69:0/2644306268 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.357+0000 7f7e337fe640 10 monclient(hunting): get_auth_request con 0x7f7e2c0c5640 auth_method 0 2023-12-15T17:02:02.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.357+0000 7f7e337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.357+0000 7f7e337fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.357+0000 7f7e337fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.357+0000 7f5ef4c09640 10 monclient: monmap: 2023-12-15T17:02:02.387 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.387 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.387 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.358746+0000 2023-12-15T17:02:02.387 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.358746+0000 2023-12-15T17:02:02.388 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.388 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.388 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.388 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.388 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.388 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.388 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.388 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.358+0000 7fb357fff640 1 --2- 172.21.15.69:0/1443912458 >> v2:172.21.15.192:3300/0 conn(0x7fb3580a8a30 0x7fb3581424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.358+0000 7f5ef4c09640 1 Processor -- start 2023-12-15T17:02:02.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.358+0000 7f5ef4c09640 1 -- start start 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.358+0000 7f5ef4c09640 10 monclient: init 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00a8a30 0x7f5ef01424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef00a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ef00a8e10 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ef0142a20 0x7f5ef0145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef0142a20 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef0142eb0 con 0x7f5ef00a8e10 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef00aaa10 con 0x7f5ef0142a20 2023-12-15T17:02:02.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00aac10 con 0x7f5ef00a8a30 2023-12-15T17:02:02.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f5ef4c09640 10 monclient(hunting): authenticate will time out at 1426.675685s 2023-12-15T17:02:02.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f4ca8ff9640 1 -- 172.21.15.69:0/1189900783 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca400ce70 con 0x7f4cac13b4c0 2023-12-15T17:02:02.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f4ca8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f4ca8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:02.392 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f4ca8ff9640 10 monclient: dump: 2023-12-15T17:02:02.392 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.392 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.392 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.392 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.392 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.393 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.393 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.393 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.393 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.393 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f4ca8ff9640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f4ca8ff9640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f4ca8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f4ca8ff9640 1 -- 172.21.15.69:0/1189900783 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca4014440 con 0x7f4cac13b4c0 2023-12-15T17:02:02.393 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f4ca8ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7fb35cf19640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2023-12-15T17:02:02.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7fb35cf19640 1 -- 172.21.15.69:0/1443912458 >> v2:172.21.15.192:3300/0 conn(0x7fb3580a8a30 msgr2=0x7fb3581424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7fb35cf19640 1 --2- 172.21.15.69:0/1443912458 >> v2:172.21.15.192:3300/0 conn(0x7fb3580a8a30 0x7fb3581424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7fb35cf19640 1 -- 172.21.15.69:0/1443912458 >> v1:172.21.15.192:6789/0 conn(0x7fb3580a8e10 legacy=0x7fb3581454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7fb35cf19640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7fb35cf19640 1 monclient: found mon.noname-b 2023-12-15T17:02:02.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7fb35cf19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7fb35cf19640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.394 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7fb35cf19640 1 -- 172.21.15.69:0/1443912458 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3580aaef0 con 0x7fb358142a20 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.359+0000 7f4ca9ffb640 4 set_mon_vals no callback set 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4cb1787640 5 monclient: authenticate success, global_id 4281 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4ca8ff9640 1 -- 172.21.15.69:0/1189900783 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4014810 con 0x7f4cac13b4c0 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4ca8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4ca8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4ca8ff9640 10 monclient: dump: 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.395 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4cb1787640 10 monclient: _renew_subs 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4cb1787640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4cb1787640 1 -- 172.21.15.69:0/1189900783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cac132c50 con 0x7f4cac13b4c0 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4cb1787640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4cb1787640 10 monclient: _renew_subs 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4cb1787640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4cb1787640 1 -- 172.21.15.69:0/1189900783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cac1330e0 con 0x7f4cac13b4c0 2023-12-15T17:02:02.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.360+0000 7f4cb1787640 1 librados: init done 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 1 -- 172.21.15.69:0/3273540366 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459800a5f0 con 0x7f45a00adb40 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 10 monclient: dump: 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.397 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 1 -- 172.21.15.69:0/3273540366 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4598033050 con 0x7f45a00adb40 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 5 monclient: authenticate success, global_id 4303 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 1 -- 172.21.15.69:0/3273540366 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459800a9b0 con 0x7f45a00adb40 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 4 set_mon_vals no callback set 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a529f640 10 monclient: dump: 2023-12-15T17:02:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.399 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f4cb1787640 10 client.4281.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f4cb1787640 15 client.4281.objecter _session_command_op_assign -1 1 2023-12-15T17:02:02.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f4cb1787640 20 client.4281.objecter _get_session osd=-1 returning homeless 2023-12-15T17:02:02.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f4cb1787640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f4cb1787640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f4cb1787640 1 -- 172.21.15.69:0/1189900783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4cac0b3ed0 con 0x7f4cac13b4c0 2023-12-15T17:02:02.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 10 monclient: shutdown 2023-12-15T17:02:02.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 1 -- 172.21.15.69:0/3273540366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45a00adb40 msgr2=0x7f45a006f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 1 --2- 172.21.15.69:0/3273540366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45a00adb40 0x7f45a006f790 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45a00adad0 tx=0x7f45980303d0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 1 -- 172.21.15.69:0/3273540366 shutdown_connections 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 1 --2- 172.21.15.69:0/3273540366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45a00adb40 0x7f45a006f790 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 1 --2- 172.21.15.69:0/3273540366 >> v2:172.21.15.69:3300/0 conn(0x7f45a013b800 0x7f45a0072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.361+0000 7f45a8505640 1 -- 172.21.15.69:0/3273540366 >> 172.21.15.69:0/3273540366 conn(0x7f45a00b01e0 msgr2=0x7f45a00b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.362+0000 7f45a8505640 1 -- 172.21.15.69:0/3273540366 shutdown_connections 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.362+0000 7f45a8505640 1 -- 172.21.15.69:0/3273540366 wait complete. 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.362+0000 7f45a8505640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.362+0000 7f45a8505640 10 monclient: monmap: 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.402 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.362+0000 7f45a8505640 1 librados: starting msgr at 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.362+0000 7f45a8505640 1 librados: starting objecter 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 1 Processor -- start 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 1 -- start start 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 1 librados: setting wanted keys 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 1 librados: calling monclient init 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 10 monclient: init 2023-12-15T17:02:02.403 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f80fa59c640 1 --2- 172.21.15.69:0/1918372231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f4038d80 0x7f80f40c5230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0097a0 tx=0x7f80ec02b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=506dffdc89bb2456 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:02:02.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45a00adb40 0x7f45a013e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 10 monclient(hunting): picked mon.noname-c con 0x7f45a00adb40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45a00af360 0x7f45a013f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 10 monclient(hunting): picked mon.noname-b con 0x7f45a00af360 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45a013b800 0x7f45a01419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 10 monclient(hunting): picked mon.noname-a con 0x7f45a013b800 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a01d0a80 con 0x7f45a013b800 2023-12-15T17:02:02.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a013dd10 con 0x7f45a00af360 2023-12-15T17:02:02.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7fb35cf19640 1 --2- 172.21.15.69:0/1443912458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb358142a20 0x7fb358145bb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb350009e50 tx=0x7fb35002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6dc609fa1295bf5 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a0141440 con 0x7f45a00adb40 2023-12-15T17:02:02.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.363+0000 7f45a8505640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.364+0000 7f45a8505640 10 monclient(hunting): authenticate will time out at 1426.680374s 2023-12-15T17:02:02.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.364+0000 7f7e33fff640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2c034540 0x7f7e2c02ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.364+0000 7f5eeed9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ef0142a20 0x7f5ef0145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.364+0000 7f7e337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.364+0000 7f7e337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.364+0000 7f7e337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.365+0000 7f7e33fff640 10 monclient(hunting): get_auth_request con 0x7f7e2c034540 auth_method 0 2023-12-15T17:02:02.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.365+0000 7f7e33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.365+0000 7f7e33fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.365+0000 7f7e33fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 1 -- 172.21.15.69:0/2808440034 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71480052d0 con 0x7f71541cabd0 2023-12-15T17:02:02.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:02.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 10 monclient: dump: 2023-12-15T17:02:02.409 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.409 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f7158f4e640 5 monclient: authenticate success, global_id 4312 2023-12-15T17:02:02.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f7158f4e640 10 monclient: _renew_subs 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f7158f4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f7158f4e640 1 -- 172.21.15.69:0/2808440034 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71541ca6f0 con 0x7f71541cabd0 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f7158f4e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f7158f4e640 10 monclient: _renew_subs 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f7158f4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f7158f4e640 1 -- 172.21.15.69:0/2808440034 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71541efb90 con 0x7f71541cabd0 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f7158f4e640 1 librados: init done 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 1 -- 172.21.15.69:0/2808440034 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7148005450 con 0x7f71541cabd0 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 1 -- 172.21.15.69:0/2808440034 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148003830 con 0x7f71541cabd0 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f713bfff640 10 monclient: dump: 2023-12-15T17:02:02.411 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.366+0000 7f7151ffb640 4 set_mon_vals no callback set 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.367+0000 7f7158f4e640 10 client.4312.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.367+0000 7f7158f4e640 15 client.4312.objecter _session_command_op_assign -1 1 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.367+0000 7f7158f4e640 20 client.4312.objecter _get_session osd=-1 returning homeless 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.367+0000 7f7158f4e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.367+0000 7f7158f4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.367+0000 7f7158f4e640 1 -- 172.21.15.69:0/2808440034 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71340028a0 con 0x7f71541cabd0 2023-12-15T17:02:02.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.368+0000 7f7e337fe640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2023-12-15T17:02:02.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.368+0000 7f7e337fe640 1 -- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e2c0c60d0 msgr2=0x7f7e2c029100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.368+0000 7f7e337fe640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e2c0c60d0 0x7f7e2c029100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.368+0000 7f7e337fe640 1 -- 172.21.15.69:0/2644306268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2c034540 msgr2=0x7f7e2c02ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.368+0000 7f7e337fe640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2c034540 0x7f7e2c02ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.368+0000 7f7e337fe640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.368+0000 7f7e337fe640 1 monclient: found mon.noname-c 2023-12-15T17:02:02.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.368+0000 7f7e337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.414 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.368+0000 7f7e337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.414 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.368+0000 7f7e337fe640 1 -- 172.21.15.69:0/2644306268 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e200094d0 con 0x7f7e2c0c5640 2023-12-15T17:02:02.414 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.368+0000 7f45a5aa0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45a00af360 0x7f45a013f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.414 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.369+0000 7f45a5aa0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45a00af360 0x7f45a013f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37222/0 (socket says 172.21.15.69:37222) 2023-12-15T17:02:02.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.369+0000 7f45a5aa0640 1 -- 172.21.15.69:0/2988014089 learned_addr learned my addr 172.21.15.69:0/2988014089 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.369+0000 7f45a5aa0640 10 monclient(hunting): get_auth_request con 0x7f45a00af360 auth_method 0 2023-12-15T17:02:02.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.369+0000 7f45a5aa0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.369+0000 7f45a5aa0640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.369+0000 7f45a5aa0640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.370+0000 7f45a5aa0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.370+0000 7f45a5aa0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.370+0000 7f45a5aa0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.372+0000 7f7e337fe640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c0c5640 0x7f7e2c15bda0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e20000c00 tx=0x7f7e200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.372+0000 7f7e30ff9640 10 client.?.objecter ms_handle_connect 0x7f7e2c0c5640 2023-12-15T17:02:02.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.372+0000 7f7e30ff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:02.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 1 -- 172.21.15.69:0/2111235687 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8031520 con 0x7fdbbc0b0f20 2023-12-15T17:02:02.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:02.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 10 monclient: dump: 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.417 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbc20ad640 5 monclient: authenticate success, global_id 4268 2023-12-15T17:02:02.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbc20ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 1 -- 172.21.15.69:0/2111235687 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb8031680 con 0x7fdbbc0b0f20 2023-12-15T17:02:02.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbc20ad640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbc20ad640 4 set_mon_vals no callback set 2023-12-15T17:02:02.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7f45a5aa0640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2023-12-15T17:02:02.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 1 -- 172.21.15.69:0/2111235687 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8031a10 con 0x7fdbbc0b0f20 2023-12-15T17:02:02.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.418 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:02.419 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7fdbb3fff640 10 monclient: dump: 2023-12-15T17:02:02.419 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.419 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.419 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.419 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.419 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.419 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.419 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.419 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.419 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.419 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7f45a5aa0640 1 -- 172.21.15.69:0/2988014089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45a00adb40 msgr2=0x7f45a013e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7f45a5aa0640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45a00adb40 0x7f45a013e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7f45a5aa0640 1 -- 172.21.15.69:0/2988014089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45a013b800 msgr2=0x7f45a01419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7f45a5aa0640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45a013b800 0x7f45a01419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7f45a5aa0640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7f45a5aa0640 1 monclient: found mon.noname-b 2023-12-15T17:02:02.420 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7f45a5aa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7f45a5aa0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.373+0000 7f45a5aa0640 1 -- 172.21.15.69:0/2988014089 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459800ca80 con 0x7f45a00af360 2023-12-15T17:02:02.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.374+0000 7fdbc20ad640 10 monclient: shutdown 2023-12-15T17:02:02.421 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.374+0000 7fdbc20ad640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.374+0000 7fdbc20ad640 1 -- 172.21.15.69:0/2111235687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc0b0f20 msgr2=0x7fdbbc0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.374+0000 7fdbc20ad640 1 --2- 172.21.15.69:0/2111235687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc0b0f20 0x7fdbbc0b2110 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80098a0 tx=0x7fdbb802b7c0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.374+0000 7fdbc20ad640 1 -- 172.21.15.69:0/2111235687 shutdown_connections 2023-12-15T17:02:02.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.374+0000 7fdbc20ad640 1 --2- 172.21.15.69:0/2111235687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc0b0f20 0x7fdbbc0b2110 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.374+0000 7fdbc20ad640 1 -- 172.21.15.69:0/2111235687 >> 172.21.15.69:0/2111235687 conn(0x7fdbbc0618d0 msgr2=0x7fdbbc0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.374+0000 7fdbc20ad640 1 -- 172.21.15.69:0/2111235687 shutdown_connections 2023-12-15T17:02:02.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.374+0000 7fdbc20ad640 1 -- 172.21.15.69:0/2111235687 wait complete. 2023-12-15T17:02:02.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.374+0000 7f5eedd9b640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f5ef00a8e10 0x7f5ef01454a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:60544/0 (socket says 172.21.15.69:60544) 2023-12-15T17:02:02.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.374+0000 7f5eedd9b640 1 -- 172.21.15.69:0/2311490771 learned_addr learned my addr 172.21.15.69:0/2311490771 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:02.423 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.375+0000 7fdbc20ad640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.423 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.375+0000 7fdbc20ad640 10 monclient: monmap: 2023-12-15T17:02:02.423 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.423 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.423 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.375+0000 7fdbc20ad640 1 librados: starting msgr at 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.375+0000 7fdbc20ad640 1 librados: starting objecter 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.375+0000 7fdbc20ad640 1 Processor -- start 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.375+0000 7fdbc20ad640 1 -- start start 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.375+0000 7fdbc20ad640 1 librados: setting wanted keys 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.375+0000 7fdbc20ad640 1 librados: calling monclient init 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.375+0000 7fdbc20ad640 10 monclient: init 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:02:02.424 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbbc0b0f20 0x7fdbbc141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 10 monclient(hunting): picked mon.noname-b con 0x7fdbbc0b0f20 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc1468b0 0x7fdbbc141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 10 monclient(hunting): picked mon.noname-c con 0x7fdbbc1468b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbbc146cd0 0x7fdbbc141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 10 monclient(hunting): picked mon.noname-a con 0x7fdbbc146cd0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1409b0 con 0x7fdbbc146cd0 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1424f0 con 0x7fdbbc0b0f20 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1427a0 con 0x7fdbbc1468b0 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7fdbc20ad640 10 monclient(hunting): authenticate will time out at 1426.692654s 2023-12-15T17:02:02.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7f4596ffd640 10 client.?.objecter ms_handle_connect 0x7f45a00af360 2023-12-15T17:02:02.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7f4596ffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:02.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.376+0000 7f45a5aa0640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45a00af360 0x7f45a013f240 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f459c00c9c0 tx=0x7f459c00ce80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.380+0000 7fb356ffd640 1 -- 172.21.15.69:0/1443912458 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35002cb00 con 0x7fb358142a20 2023-12-15T17:02:02.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.380+0000 7fb356ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.380+0000 7fb356ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.380+0000 7fb356ffd640 10 monclient: dump: 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.380+0000 7fb356ffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.380+0000 7fb356ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.380+0000 7fb356ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.380+0000 7fb35e97c640 5 monclient: authenticate success, global_id 4277 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.380+0000 7fb35e97c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.384+0000 7fb356ffd640 1 -- 172.21.15.69:0/1443912458 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35002cc60 con 0x7fb358142a20 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.384+0000 7fb356ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.384+0000 7fb356ffd640 1 -- 172.21.15.69:0/1443912458 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3500343f0 con 0x7fb358142a20 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.384+0000 7fb356ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.384+0000 7fb356ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.384+0000 7fb356ffd640 10 monclient: dump: 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.429 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.384+0000 7fb35e97c640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.384+0000 7fb35e97c640 4 set_mon_vals no callback set 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.385+0000 7fb35e97c640 10 monclient: shutdown 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.385+0000 7fb35e97c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.430 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.385+0000 7fb35e97c640 1 -- 172.21.15.69:0/1443912458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb358142a20 msgr2=0x7fb358145bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.385+0000 7fb35e97c640 1 --2- 172.21.15.69:0/1443912458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb358142a20 0x7fb358145bb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb350009e50 tx=0x7fb35002bf60 comp rx=0 tx=0).stop 2023-12-15T17:02:02.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.385+0000 7fb35e97c640 1 -- 172.21.15.69:0/1443912458 shutdown_connections 2023-12-15T17:02:02.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.385+0000 7fb35e97c640 1 --2- 172.21.15.69:0/1443912458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb358142a20 0x7fb358145bb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.385+0000 7fb35e97c640 1 --2- 172.21.15.69:0/1443912458 >> v2:172.21.15.192:3300/0 conn(0x7fb3580a8a30 0x7fb3581424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.385+0000 7fb35e97c640 1 -- 172.21.15.69:0/1443912458 >> 172.21.15.69:0/1443912458 conn(0x7fb358061a30 msgr2=0x7fb3580b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.385+0000 7fb35e97c640 1 -- 172.21.15.69:0/1443912458 shutdown_connections 2023-12-15T17:02:02.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.385+0000 7fb35e97c640 1 -- 172.21.15.69:0/1443912458 wait complete. 2023-12-15T17:02:02.431 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f5eee59c640 1 --2- 172.21.15.69:0/2311490771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00a8a30 0x7f5ef01424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7fb35e97c640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 1 -- 172.21.15.69:0/1918372231 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec031710 con 0x7f80f4038d80 2023-12-15T17:02:02.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:02.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 10 monclient: dump: 2023-12-15T17:02:02.432 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.432 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.432 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.432 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.432 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7fb35e97c640 10 monclient: monmap: 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.433 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.434 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.434 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.434 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.434 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.434 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 1 -- 172.21.15.69:0/1918372231 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ec02f050 con 0x7f80f4038d80 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 1 -- 172.21.15.69:0/1918372231 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec031a90 con 0x7f80f4038d80 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80f959a640 10 monclient: dump: 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80fb59e640 5 monclient: authenticate success, global_id 4274 2023-12-15T17:02:02.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80fb59e640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.386+0000 7f80fb59e640 4 set_mon_vals no callback set 2023-12-15T17:02:02.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f5eee59c640 10 monclient(hunting): get_auth_request con 0x7f5ef00a8a30 auth_method 0 2023-12-15T17:02:02.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f5eee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f5eee59c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f5eee59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.436 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7fb35e97c640 1 librados: starting msgr at 2023-12-15T17:02:02.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7fb35e97c640 1 librados: starting objecter 2023-12-15T17:02:02.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7fb35e97c640 1 Processor -- start 2023-12-15T17:02:02.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f80fb59e640 10 monclient: shutdown 2023-12-15T17:02:02.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f80fb59e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f80fb59e640 1 -- 172.21.15.69:0/1918372231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f4038d80 msgr2=0x7f80f40c5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f80fb59e640 1 --2- 172.21.15.69:0/1918372231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f4038d80 0x7f80f40c5230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0097a0 tx=0x7f80ec02b7b0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f80fb59e640 1 -- 172.21.15.69:0/1918372231 shutdown_connections 2023-12-15T17:02:02.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f80fb59e640 1 --2- 172.21.15.69:0/1918372231 >> v2:172.21.15.69:3300/0 conn(0x7f80f40c5770 0x7f80f40c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f80fb59e640 1 --2- 172.21.15.69:0/1918372231 >> v2:172.21.15.192:3300/0 conn(0x7f80f4039160 0x7f80f40c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f80fb59e640 1 --2- 172.21.15.69:0/1918372231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f4038d80 0x7f80f40c5230 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f80fb59e640 1 -- 172.21.15.69:0/1918372231 >> 172.21.15.69:0/1918372231 conn(0x7f80f40065e0 msgr2=0x7f80f40069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f80fb59e640 1 -- 172.21.15.69:0/1918372231 shutdown_connections 2023-12-15T17:02:02.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7f80fb59e640 1 -- 172.21.15.69:0/1918372231 wait complete. 2023-12-15T17:02:02.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7fb35e97c640 1 -- start start 2023-12-15T17:02:02.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7fb35e97c640 1 librados: setting wanted keys 2023-12-15T17:02:02.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7fb35e97c640 1 librados: calling monclient init 2023-12-15T17:02:02.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.387+0000 7fb35e97c640 10 monclient: init 2023-12-15T17:02:02.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.388+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.388+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.388+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.388+0000 7fb35e97c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.388+0000 7fb35e97c640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:02:02.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.388+0000 7fb35e97c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3580a8a30 0x7fb3580ac140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.388+0000 7fb35e97c640 10 monclient(hunting): picked mon.noname-a con 0x7fb3580a8a30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.388+0000 7fb35e97c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3580a8e10 0x7fb3581d7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.388+0000 7fb35e97c640 10 monclient(hunting): picked mon.noname-b con 0x7fb3580a8e10 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.388+0000 7fb35e97c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb358142a20 0x7fb3581d78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7fb35e97c640 10 monclient(hunting): picked mon.noname-c con 0x7fb358142a20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7fb35e97c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7fb35e97c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580ae070 con 0x7fb3580a8a30 2023-12-15T17:02:02.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7fb35e97c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7fb35e97c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580ae2c0 con 0x7fb3580a8e10 2023-12-15T17:02:02.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7fb35e97c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7fb35e97c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580acf90 con 0x7fb358142a20 2023-12-15T17:02:02.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7fb35e97c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7fb35e97c640 10 monclient(hunting): authenticate will time out at 1426.705438s 2023-12-15T17:02:02.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f80fb59e640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f80fb59e640 10 monclient: monmap: 2023-12-15T17:02:02.442 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.442 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.442 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.442 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.442 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.442 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.442 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.442 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.442 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.443 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7fdbc10ab640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbbc0b0f20 0x7fdbbc141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 1 -- 172.21.15.69:0/2644306268 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20032880 con 0x7f7e2c0c5640 2023-12-15T17:02:02.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:02.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 10 monclient: dump: 2023-12-15T17:02:02.444 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.444 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.444 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.444 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.444 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 1 -- 172.21.15.69:0/2644306268 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e200329e0 con 0x7f7e2c0c5640 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 1 -- 172.21.15.69:0/2644306268 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20032d90 con 0x7f7e2c0c5640 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e30ff9640 10 monclient: dump: 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e390aa640 5 monclient: authenticate success, global_id 4280 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e390aa640 10 monclient: _renew_subs 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e390aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e390aa640 1 -- 172.21.15.69:0/2644306268 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c0baf60 con 0x7f7e2c0c5640 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e390aa640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e390aa640 10 monclient: _renew_subs 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e390aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e390aa640 1 -- 172.21.15.69:0/2644306268 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c029820 con 0x7f7e2c0c5640 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e390aa640 1 librados: init done 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f80fb59e640 1 librados: starting msgr at 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f80fb59e640 1 librados: starting objecter 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f80fb59e640 1 Processor -- start 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f80fb59e640 1 -- start start 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f80fb59e640 1 librados: setting wanted keys 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f80fb59e640 1 librados: calling monclient init 2023-12-15T17:02:02.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f80fb59e640 10 monclient: init 2023-12-15T17:02:02.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.389+0000 7f7e327fc640 4 set_mon_vals no callback set 2023-12-15T17:02:02.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f7e390aa640 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f7e390aa640 15 client.4280.objecter _session_command_op_assign -1 1 2023-12-15T17:02:02.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f7e390aa640 20 client.4280.objecter _get_session osd=-1 returning homeless 2023-12-15T17:02:02.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f7e390aa640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f7e390aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f7e390aa640 1 -- 172.21.15.69:0/2644306268 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e2c0c6a50 con 0x7f7e2c0c5640 2023-12-15T17:02:02.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7fdbc10ab640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbbc0b0f20 0x7fdbbc141fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37224/0 (socket says 172.21.15.69:37224) 2023-12-15T17:02:02.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7fdbc10ab640 1 -- 172.21.15.69:0/2134763422 learned_addr learned my addr 172.21.15.69:0/2134763422 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7fdbc10ab640 10 monclient(hunting): get_auth_request con 0x7fdbbc0b0f20 auth_method 0 2023-12-15T17:02:02.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7fdbc10ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7fdbc10ab640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7fdbc10ab640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 1 -- 172.21.15.69:0/2988014089 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c012070 con 0x7f45a00af360 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 10 monclient: dump: 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.450 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 1 -- 172.21.15.69:0/2988014089 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459c007c60 con 0x7f45a00af360 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 1 -- 172.21.15.69:0/2988014089 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c00a420 con 0x7f45a00af360 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f4596ffd640 10 monclient: dump: 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.452 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.452 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.452 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.452 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.452 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f5eed59a640 1 -- 172.21.15.69:0/2311490771 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348351061 0 0) 0x7f5ee4001110 con 0x7f5ef00a8e10 2023-12-15T17:02:02.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f5eed59a640 10 monclient(hunting): _negotiate 2023-12-15T17:02:02.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f5eed59a640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f5eed59a640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f5eed59a640 10 monclient(hunting): my global_id is 4321 2023-12-15T17:02:02.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f5eed59a640 1 -- 172.21.15.69:0/2311490771 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed8003520 con 0x7f5ef00a8e10 2023-12-15T17:02:02.453 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.453 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:02:02.453 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f4ca8ff9640 1 -- 172.21.15.69:0/1189900783 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f4ca401d3b0 con 0x7f4cac13b4c0 2023-12-15T17:02:02.454 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80f4038d80 0x7f80f40cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.454 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f80f4038d80 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.454 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f40c5770 0x7f80f41605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.454 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f80f40c5770 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.454 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.390+0000 7f45a4a9e640 4 set_mon_vals no callback set 2023-12-15T17:02:02.454 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f45a8505640 5 monclient: authenticate success, global_id 4318 2023-12-15T17:02:02.454 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f45a8505640 10 monclient: _renew_subs 2023-12-15T17:02:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f45a8505640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f45a8505640 1 -- 172.21.15.69:0/2988014089 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a013fdf0 con 0x7f45a00af360 2023-12-15T17:02:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f45a8505640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f45a8505640 10 monclient: _renew_subs 2023-12-15T17:02:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f45a8505640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f45a8505640 1 -- 172.21.15.69:0/2988014089 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a01d6540 con 0x7f45a00af360 2023-12-15T17:02:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f45a8505640 1 librados: init done 2023-12-15T17:02:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f4ca8ff9640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f4c9806cba0 0x7f4c9806f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f4032b00 0x7f80f4032f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f80f4032b00 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f4032170 con 0x7f80f40c5770 2023-12-15T17:02:02.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f4cabfff640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f4c9806cba0 0x7f4c9806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f4cabfff640 10 monclient: get_auth_request con 0x7f4c9806cba0 auth_method 0 2023-12-15T17:02:02.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.391+0000 7f80fb59e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f40cb110 con 0x7f80f4038d80 2023-12-15T17:02:02.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f80fb59e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f80fb59e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f40c9160 con 0x7f80f4032b00 2023-12-15T17:02:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f80fb59e640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f80fb59e640 10 monclient(hunting): authenticate will time out at 1426.708606s 2023-12-15T17:02:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f5eeed9d640 10 monclient(hunting): get_auth_request con 0x7f5ef0142a20 auth_method 0 2023-12-15T17:02:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f5eeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f5eeed9d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f5eeed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f4594ff9640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f4594ff9640 15 client.4318.objecter _session_command_op_assign -1 1 2023-12-15T17:02:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f4594ff9640 20 client.4318.objecter _get_session osd=-1 returning homeless 2023-12-15T17:02:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f4594ff9640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f4594ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f4594ff9640 1 -- 172.21.15.69:0/2988014089 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f457c0028a0 con 0x7f45a00af360 2023-12-15T17:02:02.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f4cabfff640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f4c9806cba0 0x7f4c9806f060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009710 tx=0x7f4c9c006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.392+0000 7f4ca8ff9640 10 client.4281.objecter ms_handle_connect 0x7f4c9806cba0 2023-12-15T17:02:02.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eee59c640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2023-12-15T17:02:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eee59c640 1 -- 172.21.15.69:0/2311490771 >> v2:172.21.15.69:3300/0 conn(0x7f5ef0142a20 msgr2=0x7f5ef0145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eee59c640 1 --2- 172.21.15.69:0/2311490771 >> v2:172.21.15.69:3300/0 conn(0x7f5ef0142a20 0x7f5ef0145bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eee59c640 1 -- 172.21.15.69:0/2311490771 >> v1:172.21.15.192:6789/0 conn(0x7f5ef00a8e10 legacy=0x7f5ef01454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eee59c640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eee59c640 1 monclient: found mon.noname-b 2023-12-15T17:02:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eee59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.394+0000 7f5eee59c640 1 -- 172.21.15.69:0/2311490771 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef00aaef0 con 0x7f5ef00a8a30 2023-12-15T17:02:02.461 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.396+0000 7fdbc08aa640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc1468b0 0x7fdbbc141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.461 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.397+0000 7fdbc08aa640 10 monclient(hunting): get_auth_request con 0x7fdbbc1468b0 auth_method 0 2023-12-15T17:02:02.461 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.397+0000 7fdbc08aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.461 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.397+0000 7fdbc08aa640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.461 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.397+0000 7fdbc08aa640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.461 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.397+0000 7f5eee59c640 1 --2- 172.21.15.69:0/2311490771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00a8a30 0x7f5ef01424e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00097d0 tx=0x7f5ee002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afd097158f9e6e3f server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 1 -- 172.21.15.69:0/1189900783 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4ca4058da0 con 0x7f4cac13b4c0 2023-12-15T17:02:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 10 client.4281.objecter ms_dispatch 0x7f4cac000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 3 client.4281.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 3 client.4281.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 10 client.4281.objecter checking command 1 2023-12-15T17:02:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] conn(0x7f4c98072f80 0x7f4c980753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 20 client.4281.objecter _get_session s=0x7f4c980728b0 osd=7 3 2023-12-15T17:02:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 20 client.4281.objecter put_session s=0x7f4c980728b0 osd=7 3 2023-12-15T17:02:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 15 client.4281.objecter _session_command_op_remove -1 1 2023-12-15T17:02:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 20 client.4281.objecter _get_session s=0x7f4c980728b0 osd=7 3 2023-12-15T17:02:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 20 client.4281.objecter get_session s=0x7f4c980728b0 osd=7 3 2023-12-15T17:02:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 15 client.4281.objecter _session_command_op_assign 7 1 2023-12-15T17:02:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 20 client.4281.objecter put_session s=0x7f4c980728b0 osd=7 4 2023-12-15T17:02:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 10 client.4281.objecter _send_command 1 2023-12-15T17:02:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 1 -- 172.21.15.69:0/1189900783 --> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c98075a50 con 0x7f4c98072f80 2023-12-15T17:02:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 20 client.4281.objecter dump_active .. 0 homeless 2023-12-15T17:02:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 1 -- 172.21.15.69:0/1189900783 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ca40550e0 con 0x7f4cac13b4c0 2023-12-15T17:02:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7f4ca8ff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-15T17:02:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7fdbc08aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7fdbc08aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.398+0000 7fdbc08aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.399+0000 7f4cab7fe640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] conn(0x7f4c98072f80 0x7f4c980753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.399+0000 7f4cab7fe640 10 monclient: get_auth_request con 0x7f4c98072f80 auth_method 0 2023-12-15T17:02:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.401+0000 7fdbc08aa640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2023-12-15T17:02:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.401+0000 7fdbc08aa640 1 -- 172.21.15.69:0/2134763422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbbc0b0f20 msgr2=0x7fdbbc141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.401+0000 7fdbc08aa640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbbc0b0f20 0x7fdbbc141fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.401+0000 7fdbc08aa640 1 -- 172.21.15.69:0/2134763422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbbc146cd0 msgr2=0x7fdbbc141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.401+0000 7fdbc08aa640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbbc146cd0 0x7fdbbc141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.401+0000 7fdbc08aa640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.401+0000 7fdbc08aa640 1 monclient: found mon.noname-c 2023-12-15T17:02:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.401+0000 7fdbc08aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.401+0000 7fdbc08aa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.401+0000 7fdbc08aa640 1 -- 172.21.15.69:0/2134763422 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb80094d0 con 0x7fdbbc1468b0 2023-12-15T17:02:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.402+0000 7fb357fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3580a8a30 0x7fb3580ac140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.402+0000 7f4cab7fe640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] conn(0x7f4c98072f80 0x7f4c980753a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.402+0000 7f4ca8ff9640 10 client.4281.objecter ms_handle_connect 0x7f4c98072f80 2023-12-15T17:02:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.404+0000 7fb357fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3580a8a30 0x7fb3580ac140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58558/0 (socket says 172.21.15.69:58558) 2023-12-15T17:02:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.404+0000 7fb357fff640 1 -- 172.21.15.69:0/484565075 learned_addr learned my addr 172.21.15.69:0/484565075 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.404+0000 7fb357fff640 10 monclient(hunting): get_auth_request con 0x7fb3580a8a30 auth_method 0 2023-12-15T17:02:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.404+0000 7fb357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.404+0000 7fb357fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.404+0000 7fb357fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.405+0000 7f713bfff640 1 -- 172.21.15.69:0/2808440034 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7148003990 con 0x7f71541cabd0 2023-12-15T17:02:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.405+0000 7f713bfff640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f714006cbd0 0x7f714006f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.405+0000 7f80f9d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f40c5770 0x7f80f41605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.405+0000 7f71537fe640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f714006cbd0 0x7f714006f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.405+0000 7f71537fe640 10 monclient: get_auth_request con 0x7f714006cbd0 auth_method 0 2023-12-15T17:02:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.405+0000 7f80f9d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f40c5770 0x7f80f41605a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58570/0 (socket says 172.21.15.69:58570) 2023-12-15T17:02:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.405+0000 7f80f9d9b640 1 -- 172.21.15.69:0/870852848 learned_addr learned my addr 172.21.15.69:0/870852848 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.405+0000 7f80f9d9b640 10 monclient(hunting): get_auth_request con 0x7f80f40c5770 auth_method 0 2023-12-15T17:02:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.405+0000 7f80f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.405+0000 7f80f9d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.405+0000 7f80f9d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.406+0000 7fb357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.406+0000 7fb357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.406+0000 7fb357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.406+0000 7f713bfff640 10 client.4312.objecter ms_handle_connect 0x7f714006cbd0 2023-12-15T17:02:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.406+0000 7f71537fe640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f714006cbd0 0x7f714006f090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f714402bbd0 tx=0x7f7144030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.407+0000 7f80f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.407+0000 7f80f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.407+0000 7f80f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.407+0000 7fdbc08aa640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc1468b0 0x7fdbbc141510 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400c6d0 tx=0x7fdbb400cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.407+0000 7fdbb1ffb640 10 client.?.objecter ms_handle_connect 0x7fdbbc1468b0 2023-12-15T17:02:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.407+0000 7fdbb1ffb640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 1 -- 172.21.15.69:0/2808440034 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7148058000 con 0x7f71541cabd0 2023-12-15T17:02:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 10 client.4312.objecter ms_dispatch 0x7f7154000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 3 client.4312.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 3 client.4312.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 10 client.4312.objecter checking command 1 2023-12-15T17:02:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] conn(0x7f7140072f20 0x7f7140075340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 20 client.4312.objecter _get_session s=0x7f7140072850 osd=0 3 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 20 client.4312.objecter put_session s=0x7f7140072850 osd=0 3 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 15 client.4312.objecter _session_command_op_remove -1 1 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 20 client.4312.objecter _get_session s=0x7f7140072850 osd=0 3 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 20 client.4312.objecter get_session s=0x7f7140072850 osd=0 3 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 15 client.4312.objecter _session_command_op_assign 0 1 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 20 client.4312.objecter put_session s=0x7f7140072850 osd=0 4 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 10 client.4312.objecter _send_command 1 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 1 -- 172.21.15.69:0/2808440034 --> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71400759f0 con 0x7f7140072f20 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 20 client.4312.objecter dump_active .. 0 homeless 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 1 -- 172.21.15.69:0/2808440034 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7148007d50 con 0x7f71541cabd0 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.408+0000 7f713bfff640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.409+0000 7f7152ffd640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] conn(0x7f7140072f20 0x7f7140075340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.409+0000 7f7152ffd640 10 monclient: get_auth_request con 0x7f7140072f20 auth_method 0 2023-12-15T17:02:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 1 -- 172.21.15.69:0/2311490771 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0031800 con 0x7f5ef00a8a30 2023-12-15T17:02:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 10 monclient: dump: 2023-12-15T17:02:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 1 -- 172.21.15.69:0/2311490771 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee0031960 con 0x7f5ef00a8a30 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5ef4c09640 5 monclient: authenticate success, global_id 4283 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5ef4c09640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5ef4c09640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 1 -- 172.21.15.69:0/2311490771 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0031cf0 con 0x7f5ef00a8a30 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5ef4c09640 4 set_mon_vals no callback set 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5eed59a640 10 monclient: dump: 2023-12-15T17:02:02.476 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.476 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.476 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.476 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.476 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.476 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.476 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.476 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.476 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.476 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.409+0000 7fb357fff640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2023-12-15T17:02:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.409+0000 7fb357fff640 1 -- 172.21.15.69:0/484565075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb358142a20 msgr2=0x7fb3581d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7fb357fff640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb358142a20 0x7fb3581d78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7fb357fff640 1 -- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3580a8e10 msgr2=0x7fb3581d7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7fb357fff640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3580a8e10 0x7fb3581d7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7fb357fff640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7fb357fff640 1 monclient: found mon.noname-a 2023-12-15T17:02:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7fb357fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7fb357fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7fb357fff640 1 -- 172.21.15.69:0/484565075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb350009b30 con 0x7fb3580a8a30 2023-12-15T17:02:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5ef4c09640 10 monclient: shutdown 2023-12-15T17:02:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5ef4c09640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2311490771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00a8a30 msgr2=0x7f5ef01424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.410+0000 7f5ef4c09640 1 --2- 172.21.15.69:0/2311490771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00a8a30 0x7f5ef01424e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00097d0 tx=0x7f5ee002b8a0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.411+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2311490771 shutdown_connections 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.411+0000 7f5ef4c09640 1 --2- 172.21.15.69:0/2311490771 >> v2:172.21.15.69:3300/0 conn(0x7f5ef0142a20 0x7f5ef0145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.411+0000 7f5ef4c09640 1 --2- 172.21.15.69:0/2311490771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00a8a30 0x7f5ef01424e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.411+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2311490771 >> 172.21.15.69:0/2311490771 conn(0x7f5ef0061a30 msgr2=0x7f5ef00b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.411+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2311490771 shutdown_connections 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.411+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2311490771 wait complete. 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.412+0000 7f5ef4c09640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.412+0000 7f5ef4c09640 10 monclient: monmap: 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.480 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.413+0000 7f5ef4c09640 1 librados: starting msgr at 2023-12-15T17:02:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.413+0000 7f5ef4c09640 1 librados: starting objecter 2023-12-15T17:02:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.413+0000 7f5ef4c09640 1 Processor -- start 2023-12-15T17:02:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.413+0000 7f5ef4c09640 1 -- start start 2023-12-15T17:02:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.413+0000 7f5ef4c09640 1 librados: setting wanted keys 2023-12-15T17:02:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.413+0000 7f5ef4c09640 1 librados: calling monclient init 2023-12-15T17:02:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.413+0000 7f5ef4c09640 10 monclient: init 2023-12-15T17:02:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.413+0000 7f7152ffd640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] conn(0x7f7140072f20 0x7f7140075340 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.413+0000 7f713bfff640 10 client.4312.objecter ms_handle_connect 0x7f7140072f20 2023-12-15T17:02:02.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.414+0000 7f5ef4c09640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.414+0000 7f5ef4c09640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:02:02.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.414+0000 7f5ef4c09640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00a8a30 0x7f5ef00ac400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.414+0000 7f5ef4c09640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef00a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.414+0000 7f5ef4c09640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef00a8e10 0x7f5ef00b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f5ef4c09640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef00a8e10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f5ef4c09640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ef0142a20 0x7f5ef00b62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f5ef4c09640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef0142a20 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f5ef4c09640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f5ef4c09640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00ad740 con 0x7f5ef00a8e10 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f5ef4c09640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f5ef4c09640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef01398a0 con 0x7f5ef0142a20 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f5ef4c09640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f5ef4c09640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00ae560 con 0x7f5ef00a8a30 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f5ef4c09640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f5ef4c09640 10 monclient(hunting): authenticate will time out at 1426.732012s 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f80f9d9b640 10 monclient(hunting): handle_auth_done global_id 4296 payload 824 2023-12-15T17:02:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f80f9d9b640 1 -- 172.21.15.69:0/870852848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f4032b00 msgr2=0x7f80f4032f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f80f9d9b640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f4032b00 0x7f80f4032f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f80f9d9b640 1 -- 172.21.15.69:0/870852848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80f4038d80 msgr2=0x7f80f40cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f80f9d9b640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80f4038d80 0x7f80f40cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f80f9d9b640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f80f9d9b640 1 monclient: found mon.noname-a 2023-12-15T17:02:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f80f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f80f9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.415+0000 7f80f9d9b640 1 -- 172.21.15.69:0/870852848 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec009400 con 0x7f80f40c5770 2023-12-15T17:02:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.418+0000 7fb357fff640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3580a8a30 0x7fb3580ac140 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c7f0 tx=0x7fb34800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.418+0000 7fb354ff9640 10 client.?.objecter ms_handle_connect 0x7fb3580a8a30 2023-12-15T17:02:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.418+0000 7fb354ff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.418+0000 7f80f9d9b640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f40c5770 0x7f80f41605a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f80f000e8d0 tx=0x7f80f000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.418+0000 7f80eaffd640 10 client.?.objecter ms_handle_connect 0x7f80f40c5770 2023-12-15T17:02:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.419+0000 7f80eaffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.426+0000 7fdbb1ffb640 1 -- 172.21.15.69:0/2134763422 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4003b70 con 0x7fdbbc1468b0 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.426+0000 7fdbb1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.426+0000 7fdbb1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.426+0000 7fdbb1ffb640 10 monclient: dump: 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.487 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.426+0000 7fdbb1ffb640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.426+0000 7fdbb1ffb640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.426+0000 7fdbb1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.426+0000 7fdbc20ad640 5 monclient: authenticate success, global_id 4286 2023-12-15T17:02:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.426+0000 7fdbb1ffb640 1 -- 172.21.15.69:0/2134763422 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb4003cf0 con 0x7fdbbc1468b0 2023-12-15T17:02:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.426+0000 7fdbb1ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.426+0000 7fdbb1ffb640 1 -- 172.21.15.69:0/2134763422 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4003ea0 con 0x7fdbbc1468b0 2023-12-15T17:02:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbc20ad640 10 monclient: _renew_subs 2023-12-15T17:02:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbc20ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbc20ad640 1 -- 172.21.15.69:0/2134763422 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbbc139480 con 0x7fdbbc1468b0 2023-12-15T17:02:02.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 1 -- 172.21.15.69:0/870852848 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f000c940 con 0x7f80f40c5770 2023-12-15T17:02:02.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:02.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 10 monclient: dump: 2023-12-15T17:02:02.488 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.488 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.488 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 1 -- 172.21.15.69:0/870852848 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f000caa0 con 0x7f80f40c5770 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 1 -- 172.21.15.69:0/870852848 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f000ce10 con 0x7f80f40c5770 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80eaffd640 10 monclient: dump: 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80fb59e640 5 monclient: authenticate success, global_id 4296 2023-12-15T17:02:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80fb59e640 10 monclient: _renew_subs 2023-12-15T17:02:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80fb59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80fb59e640 1 -- 172.21.15.69:0/870852848 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f40ca6b0 con 0x7f80f40c5770 2023-12-15T17:02:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80ebfff640 4 set_mon_vals no callback set 2023-12-15T17:02:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80fb59e640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80fb59e640 10 monclient: _renew_subs 2023-12-15T17:02:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80fb59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbb1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbb1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbb1ffb640 10 monclient: dump: 2023-12-15T17:02:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbb37fe640 4 set_mon_vals no callback set 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbc20ad640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbc20ad640 10 monclient: _renew_subs 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbc20ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbc20ad640 1 -- 172.21.15.69:0/2134763422 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbbc139a50 con 0x7fdbbc1468b0 2023-12-15T17:02:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7fdbc20ad640 1 librados: init done 2023-12-15T17:02:02.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80fb59e640 1 -- 172.21.15.69:0/870852848 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f40c9d10 con 0x7f80f40c5770 2023-12-15T17:02:02.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.427+0000 7f80fb59e640 1 librados: init done 2023-12-15T17:02:02.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f5eedd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef00a8e10 0x7f5ef00b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f5eedd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef00a8e10 0x7f5ef00b5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58584/0 (socket says 172.21.15.69:58584) 2023-12-15T17:02:02.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f5eedd9b640 1 -- 172.21.15.69:0/2563722376 learned_addr learned my addr 172.21.15.69:0/2563722376 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f5eedd9b640 10 monclient(hunting): get_auth_request con 0x7f5ef00a8e10 auth_method 0 2023-12-15T17:02:02.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f5eedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f5eedd9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f5eedd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f5eee59c640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00a8a30 0x7f5ef00ac400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7fdbc20ad640 10 client.4286.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7fdbc20ad640 15 client.4286.objecter _session_command_op_assign -1 1 2023-12-15T17:02:02.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7fdbc20ad640 20 client.4286.objecter _get_session osd=-1 returning homeless 2023-12-15T17:02:02.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7fdbc20ad640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7fdbc20ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7fdbc20ad640 1 -- 172.21.15.69:0/2134763422 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdbbc071880 con 0x7fdbbc1468b0 2023-12-15T17:02:02.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f80fb59e640 10 client.4296.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f80fb59e640 15 client.4296.objecter _session_command_op_assign -1 1 2023-12-15T17:02:02.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f80fb59e640 20 client.4296.objecter _get_session osd=-1 returning homeless 2023-12-15T17:02:02.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f80fb59e640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f80fb59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.428+0000 7f80fb59e640 1 -- 172.21.15.69:0/870852848 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f80f40366a0 con 0x7f80f40c5770 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 1 -- 172.21.15.69:0/484565075 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348004220 con 0x7fb3580a8a30 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 10 monclient: dump: 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 1 -- 172.21.15.69:0/484565075 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3480043a0 con 0x7fb3580a8a30 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 1 -- 172.21.15.69:0/484565075 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348007860 con 0x7fb3580a8a30 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb354ff9640 10 monclient: dump: 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb35e97c640 5 monclient: authenticate success, global_id 4293 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb35e97c640 10 monclient: _renew_subs 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb35e97c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb35e97c640 1 -- 172.21.15.69:0/484565075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3581d7de0 con 0x7fb3580a8a30 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb3567fc640 4 set_mon_vals no callback set 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb35e97c640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb35e97c640 10 monclient: _renew_subs 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb35e97c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb35e97c640 1 -- 172.21.15.69:0/484565075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3580ac860 con 0x7fb3580a8a30 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.429+0000 7fb35e97c640 1 librados: init done 2023-12-15T17:02:02.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.430+0000 7fb35e97c640 10 client.4293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.430+0000 7fb35e97c640 15 client.4293.objecter _session_command_op_assign -1 1 2023-12-15T17:02:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.430+0000 7fb35e97c640 20 client.4293.objecter _get_session osd=-1 returning homeless 2023-12-15T17:02:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.430+0000 7fb35e97c640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.430+0000 7fb35e97c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.430+0000 7fb35e97c640 1 -- 172.21.15.69:0/484565075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb358141fc0 con 0x7fb3580a8a30 2023-12-15T17:02:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.437+0000 7f4596ffd640 1 -- 172.21.15.69:0/2988014089 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f459c00a580 con 0x7f45a00af360 2023-12-15T17:02:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.437+0000 7f4596ffd640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f458406cba0 0x7f458406f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.438+0000 7f5eeed9d640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ef0142a20 0x7f5ef00b62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.438+0000 7f45a62a1640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f458406cba0 0x7f458406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.438+0000 7f45a62a1640 10 monclient: get_auth_request con 0x7f458406cba0 auth_method 0 2023-12-15T17:02:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.438+0000 7f4596ffd640 10 client.4318.objecter ms_handle_connect 0x7f458406cba0 2023-12-15T17:02:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.438+0000 7f45a62a1640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f458406cba0 0x7f458406f060 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4590000f60 tx=0x7f45900079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.439+0000 7f5eedd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.439+0000 7f5eedd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.439+0000 7f5eedd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 1 -- 172.21.15.69:0/2988014089 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f459c0596c0 con 0x7f45a00af360 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 10 client.4318.objecter ms_dispatch 0x7f45a0000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 3 client.4318.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 3 client.4318.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 10 client.4318.objecter checking command 1 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] conn(0x7f4584072f80 0x7f45840753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 20 client.4318.objecter _get_session s=0x7f45840728b0 osd=1 3 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 20 client.4318.objecter put_session s=0x7f45840728b0 osd=1 3 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 15 client.4318.objecter _session_command_op_remove -1 1 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 20 client.4318.objecter _get_session s=0x7f45840728b0 osd=1 3 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 20 client.4318.objecter get_session s=0x7f45840728b0 osd=1 3 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 15 client.4318.objecter _session_command_op_assign 1 1 2023-12-15T17:02:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 20 client.4318.objecter put_session s=0x7f45840728b0 osd=1 4 2023-12-15T17:02:02.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 10 client.4318.objecter _send_command 1 2023-12-15T17:02:02.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 1 -- 172.21.15.69:0/2988014089 --> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4584075a50 con 0x7f4584072f80 2023-12-15T17:02:02.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.441+0000 7f4596ffd640 20 client.4318.objecter dump_active .. 0 homeless 2023-12-15T17:02:02.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.442+0000 7f5eeed9d640 10 monclient(hunting): get_auth_request con 0x7f5ef0142a20 auth_method 0 2023-12-15T17:02:02.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.442+0000 7f5eeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.442+0000 7f5eeed9d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.442+0000 7f5eeed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.442+0000 7f4596ffd640 1 -- 172.21.15.69:0/2988014089 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f459c055a00 con 0x7f45a00af360 2023-12-15T17:02:02.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.442+0000 7f4596ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-15T17:02:02.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.443+0000 7f5eeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.443+0000 7f5eeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.443+0000 7f5eeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.444+0000 7f5eedd9b640 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2023-12-15T17:02:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.444+0000 7f5eedd9b640 1 -- 172.21.15.69:0/2563722376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00a8a30 msgr2=0x7f5ef00ac400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.444+0000 7f5eedd9b640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00a8a30 0x7f5ef00ac400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.444+0000 7f5eedd9b640 1 -- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ef0142a20 msgr2=0x7f5ef00b62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.444+0000 7f5eedd9b640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ef0142a20 0x7f5ef00b62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.444+0000 7f5eedd9b640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.444+0000 7f5eedd9b640 1 monclient: found mon.noname-a 2023-12-15T17:02:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.444+0000 7f5eedd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.444+0000 7f5eedd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.444+0000 7f5eedd9b640 1 -- 172.21.15.69:0/2563722376 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee0009400 con 0x7f5ef00a8e10 2023-12-15T17:02:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.449+0000 7f7e30ff9640 1 -- 172.21.15.69:0/2644306268 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7e2004d3f0 con 0x7f7e2c0c5640 2023-12-15T17:02:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.449+0000 7f7e30ff9640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7e1806cb80 0x7f7e1806f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.449+0000 7f7e33fff640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7e1806cb80 0x7f7e1806f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.449+0000 7f7e33fff640 10 monclient: get_auth_request con 0x7f7e1806cb80 auth_method 0 2023-12-15T17:02:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.450+0000 7f7e30ff9640 10 client.4280.objecter ms_handle_connect 0x7f7e1806cb80 2023-12-15T17:02:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.450+0000 7f7e33fff640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7e1806cb80 0x7f7e1806f040 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7e28007fd0 tx=0x7f7e28009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.453+0000 7f4ca8ff9640 1 -- 172.21.15.69:0/1189900783 <== osd.7 v2:172.21.15.192:6826/4283271495 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4c98075a50 con 0x7f4c98072f80 2023-12-15T17:02:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.453+0000 7f4ca8ff9640 10 client.4281.objecter ms_dispatch 0x7f4cac000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:02:02.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.453+0000 7f4ca8ff9640 10 client.4281.objecter _finish_command 1 = system:0 2023-12-15T17:02:02.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.453+0000 7f4ca8ff9640 20 client.4281.objecter put_session s=0x7f4c980728b0 osd=7 4 2023-12-15T17:02:02.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.453+0000 7f4ca8ff9640 15 client.4281.objecter _session_command_op_remove 7 1 2023-12-15T17:02:02.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.454+0000 7f5eedd9b640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef00a8e10 0x7f5ef00b5db0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400c700 tx=0x7f5ee400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.454+0000 7f5edeffd640 10 client.?.objecter ms_handle_connect 0x7f5ef00a8e10 2023-12-15T17:02:02.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.454+0000 7f5edeffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 1 -- 172.21.15.69:0/2644306268 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7e2007aca0 con 0x7f7e2c0c5640 2023-12-15T17:02:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 10 client.4280.objecter ms_dispatch 0x7f7e34000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 3 client.4280.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 3 client.4280.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 10 client.4280.objecter checking command 1 2023-12-15T17:02:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] conn(0x7f7e18072f60 0x7f7e18075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 20 client.4280.objecter _get_session s=0x7f7e18072890 osd=4 3 2023-12-15T17:02:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 20 client.4280.objecter put_session s=0x7f7e18072890 osd=4 3 2023-12-15T17:02:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 15 client.4280.objecter _session_command_op_remove -1 1 2023-12-15T17:02:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 20 client.4280.objecter _get_session s=0x7f7e18072890 osd=4 3 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 20 client.4280.objecter get_session s=0x7f7e18072890 osd=4 3 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 15 client.4280.objecter _session_command_op_assign 4 1 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 20 client.4280.objecter put_session s=0x7f7e18072890 osd=4 4 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 10 client.4280.objecter _send_command 1 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 1 -- 172.21.15.69:0/2644306268 --> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e18075a30 con 0x7f7e18072f60 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 20 client.4280.objecter dump_active .. 0 homeless 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 1 -- 172.21.15.69:0/2644306268 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e200768e0 con 0x7f7e2c0c5640 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.463+0000 7f7e30ff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.464+0000 7f7e388a9640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] conn(0x7f7e18072f60 0x7f7e18075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.464+0000 7f7e388a9640 10 monclient: get_auth_request con 0x7f7e18072f60 auth_method 0 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.467+0000 7f7e388a9640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] conn(0x7f7e18072f60 0x7f7e18075380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.467+0000 7f7e30ff9640 10 client.4280.objecter ms_handle_connect 0x7f7e18072f60 2023-12-15T17:02:02.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.469+0000 7fb354ff9640 1 -- 172.21.15.69:0/484565075 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fb348007bd0 con 0x7fb3580a8a30 2023-12-15T17:02:02.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.469+0000 7fb354ff9640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fb34406cb50 0x7fb34406f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.469+0000 7fb3577fe640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fb34406cb50 0x7fb34406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.469+0000 7fb3577fe640 10 monclient: get_auth_request con 0x7fb34406cb50 auth_method 0 2023-12-15T17:02:02.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.470+0000 7fb354ff9640 10 client.4293.objecter ms_handle_connect 0x7fb34406cb50 2023-12-15T17:02:02.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.470+0000 7fb3577fe640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fb34406cb50 0x7fb34406f010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb3580abe10 tx=0x7fb34c007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.475+0000 7f713bfff640 1 -- 172.21.15.69:0/2808440034 <== osd.0 v2:172.21.15.69:6816/541068705 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f71400759f0 con 0x7f7140072f20 2023-12-15T17:02:02.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.475+0000 7f713bfff640 10 client.4312.objecter ms_dispatch 0x7f7154000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:02:02.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.475+0000 7f713bfff640 10 client.4312.objecter _finish_command 1 = system:0 2023-12-15T17:02:02.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.475+0000 7f713bfff640 20 client.4312.objecter put_session s=0x7f7140072850 osd=0 4 2023-12-15T17:02:02.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.475+0000 7f713bfff640 15 client.4312.objecter _session_command_op_remove 0 1 2023-12-15T17:02:02.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.482+0000 7f4cb1787640 10 client.4281.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-15T17:02:02.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.482+0000 7f4cb1787640 15 client.4281.objecter _session_command_op_assign -1 2 2023-12-15T17:02:02.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.482+0000 7f4cb1787640 20 client.4281.objecter _get_session s=0x7f4c980728b0 osd=7 3 2023-12-15T17:02:02.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.482+0000 7f4cb1787640 20 client.4281.objecter put_session s=0x7f4c980728b0 osd=7 3 2023-12-15T17:02:02.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.482+0000 7f4cb1787640 20 client.4281.objecter _get_session s=0x7f4c980728b0 osd=7 3 2023-12-15T17:02:02.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.482+0000 7f4cb1787640 15 client.4281.objecter _session_command_op_remove -1 2 2023-12-15T17:02:02.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.482+0000 7f4cb1787640 20 client.4281.objecter get_session s=0x7f4c980728b0 osd=7 3 2023-12-15T17:02:02.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.482+0000 7f4cb1787640 15 client.4281.objecter _session_command_op_assign 7 2 2023-12-15T17:02:02.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.482+0000 7f4cb1787640 20 client.4281.objecter put_session s=0x7f4c980728b0 osd=7 4 2023-12-15T17:02:02.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.482+0000 7f4cb1787640 10 client.4281.objecter _send_command 2 2023-12-15T17:02:02.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.482+0000 7f4cb1787640 1 -- 172.21.15.69:0/1189900783 --> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4cac13bfb0 con 0x7f4c98072f80 2023-12-15T17:02:02.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.483+0000 7fdbb1ffb640 1 -- 172.21.15.69:0/2134763422 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fdbb40052b0 con 0x7fdbbc1468b0 2023-12-15T17:02:02.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.484+0000 7fdbb1ffb640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fdba006cb50 0x7fdba006f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.484+0000 7fdbc10ab640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fdba006cb50 0x7fdba006f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.484+0000 7fdbc10ab640 10 monclient: get_auth_request con 0x7fdba006cb50 auth_method 0 2023-12-15T17:02:02.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.484+0000 7fdbc10ab640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fdba006cb50 0x7fdba006f010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdbb802b800 tx=0x7fdbb8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.485+0000 7fdbb1ffb640 10 client.4286.objecter ms_handle_connect 0x7fdba006cb50 2023-12-15T17:02:02.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 1 -- 172.21.15.69:0/2134763422 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdbb4058940 con 0x7fdbbc1468b0 2023-12-15T17:02:02.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 10 client.4286.objecter ms_dispatch 0x7fdbbc000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 3 client.4286.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 3 client.4286.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 10 client.4286.objecter checking command 1 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] conn(0x7fdba0072f30 0x7fdba0075350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 20 client.4286.objecter _get_session s=0x7fdba0072860 osd=3 3 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 20 client.4286.objecter put_session s=0x7fdba0072860 osd=3 3 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 15 client.4286.objecter _session_command_op_remove -1 1 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 20 client.4286.objecter _get_session s=0x7fdba0072860 osd=3 3 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 20 client.4286.objecter get_session s=0x7fdba0072860 osd=3 3 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 15 client.4286.objecter _session_command_op_assign 3 1 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 20 client.4286.objecter put_session s=0x7fdba0072860 osd=3 4 2023-12-15T17:02:02.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 10 client.4286.objecter _send_command 1 2023-12-15T17:02:02.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 1 -- 172.21.15.69:0/2134763422 --> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdba0075a00 con 0x7fdba0072f30 2023-12-15T17:02:02.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 20 client.4286.objecter dump_active .. 0 homeless 2023-12-15T17:02:02.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 1 -- 172.21.15.69:0/2134763422 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbb4001760 con 0x7fdbbc1468b0 2023-12-15T17:02:02.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.486+0000 7fdbb1ffb640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-15T17:02:02.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.487+0000 7fdbc18ac640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] conn(0x7fdba0072f30 0x7fdba0075350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.487+0000 7fdbc18ac640 10 monclient: get_auth_request con 0x7fdba0072f30 auth_method 0 2023-12-15T17:02:02.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4ca8ff9640 1 -- 172.21.15.69:0/1189900783 <== osd.7 v2:172.21.15.192:6826/4283271495 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4ca0031820 con 0x7f4c98072f80 2023-12-15T17:02:02.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4ca8ff9640 10 client.4281.objecter ms_dispatch 0x7f4cac000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:02:02.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4ca8ff9640 10 client.4281.objecter _finish_command 2 = system:0 2023-12-15T17:02:02.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4ca8ff9640 20 client.4281.objecter put_session s=0x7f4c980728b0 osd=7 4 2023-12-15T17:02:02.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4ca8ff9640 15 client.4281.objecter _session_command_op_remove 7 2 2023-12-15T17:02:02.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 10 librados: watch_flush enter 2023-12-15T17:02:02.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 10 librados: watch_flush exit 2023-12-15T17:02:02.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 10 client.4281.objecter close_session for osd.7 2023-12-15T17:02:02.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 1 -- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] conn(0x7f4c98072f80 msgr2=0x7f4c980753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] conn(0x7f4c98072f80 0x7f4c980753a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 20 client.4281.objecter put_session s=0x7f4c980728b0 osd=7 1 2023-12-15T17:02:02.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 20 client.4281.objecter shutdown clearing up homeless session... 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 10 client.4281.objecter successfully canceled tick 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 1 -- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f4c9806cba0 msgr2=0x7f4c9806f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f4c9806cba0 0x7f4c9806f060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009710 tx=0x7f4c9c006c90 comp rx=0 tx=0).stop 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 10 monclient: shutdown 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 1 -- 172.21.15.69:0/1189900783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4cac13b4c0 msgr2=0x7f4cac132710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 1 --2- 172.21.15.69:0/1189900783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4cac13b4c0 0x7f4cac132710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400be20 tx=0x7f4ca400bf10 comp rx=0 tx=0).stop 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb0f86640 1 -- 172.21.15.69:0/1189900783 reap_dead start 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 1 -- 172.21.15.69:0/1189900783 shutdown_connections 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 1 -- 172.21.15.69:0/1189900783 >> 172.21.15.69:0/1189900783 conn(0x7f4cac06ace0 msgr2=0x7f4cac1348b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 1 -- 172.21.15.69:0/1189900783 shutdown_connections 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.488+0000 7f4cb1787640 1 -- 172.21.15.69:0/1189900783 wait complete. 2023-12-15T17:02:02.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.489+0000 7f4cb1787640 1 librados: shutdown 2023-12-15T17:02:02.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7f45a6aa2640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] conn(0x7f4584072f80 0x7f45840753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7f45a6aa2640 10 monclient: get_auth_request con 0x7f4584072f80 auth_method 0 2023-12-15T17:02:02.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7f80eaffd640 1 -- 172.21.15.69:0/870852848 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f80f0013470 con 0x7f80f40c5770 2023-12-15T17:02:02.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f80eaffd640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f80dc06cbd0 0x7f80dc06f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f80fa59c640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f80dc06cbd0 0x7f80dc06f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 1 -- 172.21.15.69:0/484565075 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb348058f00 con 0x7fb3580a8a30 2023-12-15T17:02:02.524 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 10 client.4293.objecter ms_dispatch 0x7fb358000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:02.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 3 client.4293.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:02.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 3 client.4293.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:02.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 10 client.4293.objecter checking command 1 2023-12-15T17:02:02.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] conn(0x7fb344072f30 0x7fb344075350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 20 client.4293.objecter _get_session s=0x7fb344072860 osd=5 3 2023-12-15T17:02:02.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 20 client.4293.objecter put_session s=0x7fb344072860 osd=5 3 2023-12-15T17:02:02.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 15 client.4293.objecter _session_command_op_remove -1 1 2023-12-15T17:02:02.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 20 client.4293.objecter _get_session s=0x7fb344072860 osd=5 3 2023-12-15T17:02:02.525 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 20 client.4293.objecter get_session s=0x7fb344072860 osd=5 3 2023-12-15T17:02:02.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 15 client.4293.objecter _session_command_op_assign 5 1 2023-12-15T17:02:02.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 20 client.4293.objecter put_session s=0x7fb344072860 osd=5 4 2023-12-15T17:02:02.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 10 client.4293.objecter _send_command 1 2023-12-15T17:02:02.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 1 -- 172.21.15.69:0/484565075 --> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb344075a00 con 0x7fb344072f30 2023-12-15T17:02:02.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 20 client.4293.objecter dump_active .. 0 homeless 2023-12-15T17:02:02.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 1 -- 172.21.15.69:0/484565075 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb348007e80 con 0x7fb3580a8a30 2023-12-15T17:02:02.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.491+0000 7fb354ff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-15T17:02:02.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f80fa59c640 10 monclient: get_auth_request con 0x7f80dc06cbd0 auth_method 0 2023-12-15T17:02:02.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7fdbc18ac640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] conn(0x7fdba0072f30 0x7fdba0075350 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7fdbb1ffb640 10 client.4286.objecter ms_handle_connect 0x7fdba0072f30 2023-12-15T17:02:02.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f7158f4e640 10 client.4312.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-15T17:02:02.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f7158f4e640 15 client.4312.objecter _session_command_op_assign -1 2 2023-12-15T17:02:02.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f7158f4e640 20 client.4312.objecter _get_session s=0x7f7140072850 osd=0 3 2023-12-15T17:02:02.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f7158f4e640 20 client.4312.objecter put_session s=0x7f7140072850 osd=0 3 2023-12-15T17:02:02.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f7158f4e640 20 client.4312.objecter _get_session s=0x7f7140072850 osd=0 3 2023-12-15T17:02:02.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f7158f4e640 15 client.4312.objecter _session_command_op_remove -1 2 2023-12-15T17:02:02.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f7158f4e640 20 client.4312.objecter get_session s=0x7f7140072850 osd=0 3 2023-12-15T17:02:02.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f7158f4e640 15 client.4312.objecter _session_command_op_assign 0 2 2023-12-15T17:02:02.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f7158f4e640 20 client.4312.objecter put_session s=0x7f7140072850 osd=0 4 2023-12-15T17:02:02.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f7158f4e640 10 client.4312.objecter _send_command 2 2023-12-15T17:02:02.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.492+0000 7f7158f4e640 1 -- 172.21.15.69:0/2808440034 --> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f71340011f0 con 0x7f7140072f20 2023-12-15T17:02:02.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.493+0000 7f80fa59c640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f80dc06cbd0 0x7f80dc06f090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f80ec035000 tx=0x7f80ec030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.530 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.493+0000 7f80eaffd640 10 client.4296.objecter ms_handle_connect 0x7f80dc06cbd0 2023-12-15T17:02:02.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.496+0000 7f45a6aa2640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] conn(0x7f4584072f80 0x7f45840753a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.496+0000 7f4596ffd640 10 client.4318.objecter ms_handle_connect 0x7f4584072f80 2023-12-15T17:02:02.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 1 -- 172.21.15.69:0/870852848 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f80f0059040 con 0x7f80f40c5770 2023-12-15T17:02:02.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 10 client.4296.objecter ms_dispatch 0x7f80fc000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 3 client.4296.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 3 client.4296.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 10 client.4296.objecter checking command 1 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x7f80dc072fb0 0x7f80dc0753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 20 client.4296.objecter _get_session s=0x7f80dc0728e0 osd=2 3 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 20 client.4296.objecter put_session s=0x7f80dc0728e0 osd=2 3 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 15 client.4296.objecter _session_command_op_remove -1 1 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 20 client.4296.objecter _get_session s=0x7f80dc0728e0 osd=2 3 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 20 client.4296.objecter get_session s=0x7f80dc0728e0 osd=2 3 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 15 client.4296.objecter _session_command_op_assign 2 1 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 20 client.4296.objecter put_session s=0x7f80dc0728e0 osd=2 4 2023-12-15T17:02:02.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 10 client.4296.objecter _send_command 1 2023-12-15T17:02:02.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 1 -- 172.21.15.69:0/870852848 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f80dc075a80 con 0x7f80dc072fb0 2023-12-15T17:02:02.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 20 client.4296.objecter dump_active .. 0 homeless 2023-12-15T17:02:02.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 1 -- 172.21.15.69:0/870852848 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80f0054c80 con 0x7f80f40c5770 2023-12-15T17:02:02.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.499+0000 7f80eaffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-15T17:02:02.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.500+0000 7f80fad9d640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x7f80dc072fb0 0x7f80dc0753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.500+0000 7f713bfff640 1 -- 172.21.15.69:0/2808440034 <== osd.0 v2:172.21.15.69:6816/541068705 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f71340011f0 con 0x7f7140072f20 2023-12-15T17:02:02.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.500+0000 7f713bfff640 10 client.4312.objecter ms_dispatch 0x7f7154000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:02:02.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.500+0000 7f713bfff640 10 client.4312.objecter _finish_command 2 = system:0 2023-12-15T17:02:02.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.500+0000 7f713bfff640 20 client.4312.objecter put_session s=0x7f7140072850 osd=0 4 2023-12-15T17:02:02.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.500+0000 7f713bfff640 15 client.4312.objecter _session_command_op_remove 0 2 2023-12-15T17:02:02.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.500+0000 7f80fad9d640 10 monclient: get_auth_request con 0x7f80dc072fb0 auth_method 0 2023-12-15T17:02:02.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 10 librados: watch_flush enter 2023-12-15T17:02:02.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 10 librados: watch_flush exit 2023-12-15T17:02:02.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 10 client.4312.objecter close_session for osd.0 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 1 -- 172.21.15.69:0/2808440034 >> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] conn(0x7f7140072f20 msgr2=0x7f7140075340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] conn(0x7f7140072f20 0x7f7140075340 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 20 client.4312.objecter put_session s=0x7f7140072850 osd=0 1 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 20 client.4312.objecter shutdown clearing up homeless session... 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 10 client.4312.objecter successfully canceled tick 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 1 -- 172.21.15.69:0/2808440034 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f714006cbd0 msgr2=0x7f714006f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f714006cbd0 0x7f714006f090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f714402bbd0 tx=0x7f7144030000 comp rx=0 tx=0).stop 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 10 monclient: shutdown 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 1 -- 172.21.15.69:0/2808440034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71541cabd0 msgr2=0x7f71541ebd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 1 --2- 172.21.15.69:0/2808440034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71541cabd0 0x7f71541ebd50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f714800c3f0 tx=0x7f7148001e70 comp rx=0 tx=0).stop 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7153fff640 1 -- 172.21.15.69:0/2808440034 reap_dead start 2023-12-15T17:02:02.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 1 -- 172.21.15.69:0/2808440034 shutdown_connections 2023-12-15T17:02:02.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.501+0000 7f7158f4e640 1 -- 172.21.15.69:0/2808440034 >> 172.21.15.69:0/2808440034 conn(0x7f715406ab30 msgr2=0x7f71540b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5edeffd640 1 -- 172.21.15.69:0/2563722376 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4005ce0 con 0x7f5ef00a8e10 2023-12-15T17:02:02.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5edeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.537 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5edeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5edeffd640 10 monclient: dump: 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5edeffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5edeffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5edeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5ef4c09640 5 monclient: authenticate success, global_id 4299 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5ef4c09640 10 monclient: _renew_subs 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5ef4c09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2563722376 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef00ab5c0 con 0x7f5ef00a8e10 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5ef4c09640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5ef4c09640 10 monclient: _renew_subs 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5ef4c09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2563722376 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef00b6830 con 0x7f5ef00a8e10 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.502+0000 7f5ef4c09640 1 librados: init done 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.503+0000 7f5ef4c09640 10 client.4299.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.503+0000 7f5ef4c09640 15 client.4299.objecter _session_command_op_assign -1 1 2023-12-15T17:02:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.503+0000 7f5ef4c09640 20 client.4299.objecter _get_session osd=-1 returning homeless 2023-12-15T17:02:02.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.503+0000 7f5ef4c09640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.503+0000 7f5ef4c09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.503+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2563722376 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5ef0141fc0 con 0x7f5ef00a8e10 2023-12-15T17:02:02.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.503+0000 7f7158f4e640 1 -- 172.21.15.69:0/2808440034 shutdown_connections 2023-12-15T17:02:02.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.503+0000 7f7158f4e640 1 -- 172.21.15.69:0/2808440034 wait complete. 2023-12-15T17:02:02.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.504+0000 7f7158f4e640 1 librados: shutdown 2023-12-15T17:02:02.541 INFO:teuthology.orchestra.run.smithi069.stdout:60129542149 2023-12-15T17:02:02.542 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-15T17:02:02.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.505+0000 7fb35cf19640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] conn(0x7fb344072f30 0x7fb344075350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.505+0000 7f80eaffd640 10 client.4296.objecter ms_handle_connect 0x7f80dc072fb0 2023-12-15T17:02:02.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.505+0000 7f80fad9d640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x7f80dc072fb0 0x7f80dc0753d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.506+0000 7fb35cf19640 10 monclient: get_auth_request con 0x7fb344072f30 auth_method 0 2023-12-15T17:02:02.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.511+0000 7fb354ff9640 10 client.4293.objecter ms_handle_connect 0x7fb344072f30 2023-12-15T17:02:02.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.511+0000 7fb35cf19640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] conn(0x7fb344072f30 0x7fb344075350 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.515+0000 7f7e30ff9640 1 -- 172.21.15.69:0/2644306268 <== osd.4 v2:172.21.15.192:6802/1420550374 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7e18075a30 con 0x7f7e18072f60 2023-12-15T17:02:02.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.515+0000 7f7e30ff9640 10 client.4280.objecter ms_dispatch 0x7f7e34000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:02:02.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.515+0000 7f7e30ff9640 10 client.4280.objecter _finish_command 1 = system:0 2023-12-15T17:02:02.544 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.515+0000 7f7e30ff9640 20 client.4280.objecter put_session s=0x7f7e18072890 osd=4 4 2023-12-15T17:02:02.544 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.515+0000 7f7e30ff9640 15 client.4280.objecter _session_command_op_remove 4 1 2023-12-15T17:02:02.544 INFO:teuthology.orchestra.run.smithi069.stdout:64424509447 2023-12-15T17:02:02.545 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-15T17:02:02.547 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.529+0000 7f5edeffd640 1 -- 172.21.15.69:0/2563722376 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee4007da0 con 0x7f5ef00a8e10 2023-12-15T17:02:02.547 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.529+0000 7f5edeffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.547 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.529+0000 7f5edeffd640 1 -- 172.21.15.69:0/2563722376 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4007270 con 0x7f5ef00a8e10 2023-12-15T17:02:02.547 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.529+0000 7f5edeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.547 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.529+0000 7f5edeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:02.547 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.529+0000 7f5edeffd640 10 monclient: dump: 2023-12-15T17:02:02.547 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.547 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.547 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.547 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.547 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.529+0000 7f5eecd99640 4 set_mon_vals no callback set 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.530+0000 7f5edeffd640 1 -- 172.21.15.69:0/2563722376 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f5ee40074b0 con 0x7f5ef00a8e10 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.530+0000 7f5edeffd640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f5ed006cb50 0x7f5ed006f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.531+0000 7f5eee59c640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f5ed006cb50 0x7f5ed006f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.531+0000 7f5eee59c640 10 monclient: get_auth_request con 0x7f5ed006cb50 auth_method 0 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.531+0000 7f5eee59c640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f5ed006cb50 0x7f5ed006f010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0036000 tx=0x7f5ee0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.531+0000 7f5edeffd640 10 client.4299.objecter ms_handle_connect 0x7f5ed006cb50 2023-12-15T17:02:02.548 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.534+0000 7f5edeffd640 1 -- 172.21.15.69:0/2563722376 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5ee4058370 con 0x7f5ef00a8e10 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.534+0000 7f5edeffd640 10 client.4299.objecter ms_dispatch 0x7f5ef0000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.534+0000 7f5edeffd640 3 client.4299.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.534+0000 7f5edeffd640 3 client.4299.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 10 client.4299.objecter checking command 1 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] conn(0x7f5ed0072f30 0x7f5ed0075350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 20 client.4299.objecter _get_session s=0x7f5ed0072860 osd=6 3 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 20 client.4299.objecter put_session s=0x7f5ed0072860 osd=6 3 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 15 client.4299.objecter _session_command_op_remove -1 1 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 20 client.4299.objecter _get_session s=0x7f5ed0072860 osd=6 3 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 20 client.4299.objecter get_session s=0x7f5ed0072860 osd=6 3 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 15 client.4299.objecter _session_command_op_assign 6 1 2023-12-15T17:02:02.549 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 20 client.4299.objecter put_session s=0x7f5ed0072860 osd=6 4 2023-12-15T17:02:02.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 10 client.4299.objecter _send_command 1 2023-12-15T17:02:02.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 1 -- 172.21.15.69:0/2563722376 --> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ed0075a00 con 0x7f5ed0072f30 2023-12-15T17:02:02.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 20 client.4299.objecter dump_active .. 0 homeless 2023-12-15T17:02:02.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 1 -- 172.21.15.69:0/2563722376 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ee40546b0 con 0x7f5ef00a8e10 2023-12-15T17:02:02.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5edeffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-15T17:02:02.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5eeed9d640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] conn(0x7f5ed0072f30 0x7f5ed0075350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.535+0000 7f5eeed9d640 10 monclient: get_auth_request con 0x7f5ed0072f30 auth_method 0 2023-12-15T17:02:02.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.539+0000 7f5eeed9d640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] conn(0x7f5ed0072f30 0x7f5ed0075350 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.550 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.539+0000 7f5edeffd640 10 client.4299.objecter ms_handle_connect 0x7f5ed0072f30 2023-12-15T17:02:02.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.545+0000 7f7e390aa640 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-15T17:02:02.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.545+0000 7f7e390aa640 15 client.4280.objecter _session_command_op_assign -1 2 2023-12-15T17:02:02.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.545+0000 7f7e390aa640 20 client.4280.objecter _get_session s=0x7f7e18072890 osd=4 3 2023-12-15T17:02:02.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.545+0000 7f7e390aa640 20 client.4280.objecter put_session s=0x7f7e18072890 osd=4 3 2023-12-15T17:02:02.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.545+0000 7f7e390aa640 20 client.4280.objecter _get_session s=0x7f7e18072890 osd=4 3 2023-12-15T17:02:02.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.545+0000 7f7e390aa640 15 client.4280.objecter _session_command_op_remove -1 2 2023-12-15T17:02:02.551 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.545+0000 7f7e390aa640 20 client.4280.objecter get_session s=0x7f7e18072890 osd=4 3 2023-12-15T17:02:02.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.545+0000 7f7e390aa640 15 client.4280.objecter _session_command_op_assign 4 2 2023-12-15T17:02:02.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.545+0000 7f7e390aa640 20 client.4280.objecter put_session s=0x7f7e18072890 osd=4 4 2023-12-15T17:02:02.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.545+0000 7f7e390aa640 10 client.4280.objecter _send_command 2 2023-12-15T17:02:02.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.545+0000 7f7e390aa640 1 -- 172.21.15.69:0/2644306268 --> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7e2c0c6a50 con 0x7f7e18072f60 2023-12-15T17:02:02.554 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.552+0000 7f7e30ff9640 1 -- 172.21.15.69:0/2644306268 <== osd.4 v2:172.21.15.192:6802/1420550374 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7e3404f730 con 0x7f7e18072f60 2023-12-15T17:02:02.554 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.552+0000 7f7e30ff9640 10 client.4280.objecter ms_dispatch 0x7f7e34000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:02:02.554 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.552+0000 7f7e30ff9640 10 client.4280.objecter _finish_command 2 = system:0 2023-12-15T17:02:02.554 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.552+0000 7f7e30ff9640 20 client.4280.objecter put_session s=0x7f7e18072890 osd=4 4 2023-12-15T17:02:02.554 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.552+0000 7f7e30ff9640 15 client.4280.objecter _session_command_op_remove 4 2 2023-12-15T17:02:02.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.552+0000 7f7e390aa640 10 librados: watch_flush enter 2023-12-15T17:02:02.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.552+0000 7f7e390aa640 10 librados: watch_flush exit 2023-12-15T17:02:02.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 10 client.4280.objecter close_session for osd.4 2023-12-15T17:02:02.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 1 -- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] conn(0x7f7e18072f60 msgr2=0x7f7e18075380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] conn(0x7f7e18072f60 0x7f7e18075380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 20 client.4280.objecter put_session s=0x7f7e18072890 osd=4 1 2023-12-15T17:02:02.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 20 client.4280.objecter shutdown clearing up homeless session... 2023-12-15T17:02:02.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 10 client.4280.objecter successfully canceled tick 2023-12-15T17:02:02.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 1 -- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7e1806cb80 msgr2=0x7f7e1806f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7e1806cb80 0x7f7e1806f040 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7e28007fd0 tx=0x7f7e28009040 comp rx=0 tx=0).stop 2023-12-15T17:02:02.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 10 monclient: shutdown 2023-12-15T17:02:02.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 1 -- 172.21.15.69:0/2644306268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c0c5640 msgr2=0x7f7e2c15bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 1 --2- 172.21.15.69:0/2644306268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c0c5640 0x7f7e2c15bda0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e20000c00 tx=0x7f7e200046c0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e388a9640 1 -- 172.21.15.69:0/2644306268 reap_dead start 2023-12-15T17:02:02.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 1 -- 172.21.15.69:0/2644306268 shutdown_connections 2023-12-15T17:02:02.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.553+0000 7f7e390aa640 1 -- 172.21.15.69:0/2644306268 >> 172.21.15.69:0/2644306268 conn(0x7f7e2c036610 msgr2=0x7f7e2c0c46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.554+0000 7f7e390aa640 1 -- 172.21.15.69:0/2644306268 shutdown_connections 2023-12-15T17:02:02.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.554+0000 7f7e390aa640 1 -- 172.21.15.69:0/2644306268 wait complete. 2023-12-15T17:02:02.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.554+0000 7f7e390aa640 1 librados: shutdown 2023-12-15T17:02:02.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.557+0000 7fb354ff9640 1 -- 172.21.15.69:0/484565075 <== osd.5 v2:172.21.15.192:6810/2003653981 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb344075a00 con 0x7fb344072f30 2023-12-15T17:02:02.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.557+0000 7fb354ff9640 10 client.4293.objecter ms_dispatch 0x7fb358000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:02:02.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.557+0000 7fb354ff9640 10 client.4293.objecter _finish_command 1 = system:0 2023-12-15T17:02:02.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.557+0000 7fb354ff9640 20 client.4293.objecter put_session s=0x7fb344072860 osd=5 4 2023-12-15T17:02:02.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.557+0000 7fb354ff9640 15 client.4293.objecter _session_command_op_remove 5 1 2023-12-15T17:02:02.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.566+0000 7fdbb1ffb640 1 -- 172.21.15.69:0/2134763422 <== osd.3 v2:172.21.15.69:6824/882900488 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fdba0075a00 con 0x7fdba0072f30 2023-12-15T17:02:02.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.566+0000 7fdbb1ffb640 10 client.4286.objecter ms_dispatch 0x7fdbbc000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:02:02.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.566+0000 7fdbb1ffb640 10 client.4286.objecter _finish_command 1 = system:0 2023-12-15T17:02:02.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.566+0000 7fdbb1ffb640 20 client.4286.objecter put_session s=0x7fdba0072860 osd=3 4 2023-12-15T17:02:02.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.566+0000 7fdbb1ffb640 15 client.4286.objecter _session_command_op_remove 3 1 2023-12-15T17:02:02.570 INFO:teuthology.orchestra.run.smithi069.stdout:60129542151 2023-12-15T17:02:02.571 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-15T17:02:02.573 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.572+0000 7f80eaffd640 1 -- 172.21.15.69:0/870852848 <== osd.2 v2:172.21.15.69:6800/3322029776 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f80dc075a80 con 0x7f80dc072fb0 2023-12-15T17:02:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.572+0000 7f80eaffd640 10 client.4296.objecter ms_dispatch 0x7f80fc000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:02:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.572+0000 7f80eaffd640 10 client.4296.objecter _finish_command 1 = system:0 2023-12-15T17:02:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.572+0000 7f80eaffd640 20 client.4296.objecter put_session s=0x7f80dc0728e0 osd=2 4 2023-12-15T17:02:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.572+0000 7f80eaffd640 15 client.4296.objecter _session_command_op_remove 2 1 2023-12-15T17:02:02.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.581+0000 7f4596ffd640 1 -- 172.21.15.69:0/2988014089 <== osd.1 v2:172.21.15.69:6808/1683202761 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4584075a50 con 0x7f4584072f80 2023-12-15T17:02:02.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.581+0000 7f4596ffd640 10 client.4318.objecter ms_dispatch 0x7f45a0000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:02:02.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.581+0000 7f4596ffd640 10 client.4318.objecter _finish_command 1 = system:0 2023-12-15T17:02:02.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.581+0000 7f4596ffd640 20 client.4318.objecter put_session s=0x7f45840728b0 osd=1 4 2023-12-15T17:02:02.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.581+0000 7f4596ffd640 15 client.4318.objecter _session_command_op_remove 1 1 2023-12-15T17:02:02.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.585+0000 7fb35e97c640 10 client.4293.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-15T17:02:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.585+0000 7fb35e97c640 15 client.4293.objecter _session_command_op_assign -1 2 2023-12-15T17:02:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.585+0000 7fb35e97c640 20 client.4293.objecter _get_session s=0x7fb344072860 osd=5 3 2023-12-15T17:02:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.585+0000 7fb35e97c640 20 client.4293.objecter put_session s=0x7fb344072860 osd=5 3 2023-12-15T17:02:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.585+0000 7fb35e97c640 20 client.4293.objecter _get_session s=0x7fb344072860 osd=5 3 2023-12-15T17:02:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.585+0000 7fb35e97c640 15 client.4293.objecter _session_command_op_remove -1 2 2023-12-15T17:02:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.585+0000 7fb35e97c640 20 client.4293.objecter get_session s=0x7fb344072860 osd=5 3 2023-12-15T17:02:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.585+0000 7fb35e97c640 15 client.4293.objecter _session_command_op_assign 5 2 2023-12-15T17:02:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.585+0000 7fb35e97c640 20 client.4293.objecter put_session s=0x7fb344072860 osd=5 4 2023-12-15T17:02:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.585+0000 7fb35e97c640 10 client.4293.objecter _send_command 2 2023-12-15T17:02:02.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.585+0000 7fb35e97c640 1 -- 172.21.15.69:0/484565075 --> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb358067ac0 con 0x7fb344072f30 2023-12-15T17:02:02.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.587+0000 7f5edeffd640 1 -- 172.21.15.69:0/2563722376 <== osd.6 v2:172.21.15.192:6818/2563835645 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5ed0075a00 con 0x7f5ed0072f30 2023-12-15T17:02:02.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.587+0000 7f5edeffd640 10 client.4299.objecter ms_dispatch 0x7f5ef0000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:02:02.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.587+0000 7f5edeffd640 10 client.4299.objecter _finish_command 1 = system:0 2023-12-15T17:02:02.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.587+0000 7f5edeffd640 20 client.4299.objecter put_session s=0x7f5ed0072860 osd=6 4 2023-12-15T17:02:02.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.587+0000 7f5edeffd640 15 client.4299.objecter _session_command_op_remove 6 1 2023-12-15T17:02:02.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fdbc20ad640 10 client.4286.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-15T17:02:02.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fdbc20ad640 15 client.4286.objecter _session_command_op_assign -1 2 2023-12-15T17:02:02.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fdbc20ad640 20 client.4286.objecter _get_session s=0x7fdba0072860 osd=3 3 2023-12-15T17:02:02.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fdbc20ad640 20 client.4286.objecter put_session s=0x7fdba0072860 osd=3 3 2023-12-15T17:02:02.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fdbc20ad640 20 client.4286.objecter _get_session s=0x7fdba0072860 osd=3 3 2023-12-15T17:02:02.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fdbc20ad640 15 client.4286.objecter _session_command_op_remove -1 2 2023-12-15T17:02:02.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fdbc20ad640 20 client.4286.objecter get_session s=0x7fdba0072860 osd=3 3 2023-12-15T17:02:02.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fdbc20ad640 15 client.4286.objecter _session_command_op_assign 3 2 2023-12-15T17:02:02.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fdbc20ad640 20 client.4286.objecter put_session s=0x7fdba0072860 osd=3 4 2023-12-15T17:02:02.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fdbc20ad640 10 client.4286.objecter _send_command 2 2023-12-15T17:02:02.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fdbc20ad640 1 -- 172.21.15.69:0/2134763422 --> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdbbc143ca0 con 0x7fdba0072f30 2023-12-15T17:02:02.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fb354ff9640 1 -- 172.21.15.69:0/484565075 <== osd.5 v2:172.21.15.192:6810/2003653981 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb358067ac0 con 0x7fb344072f30 2023-12-15T17:02:02.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fb354ff9640 10 client.4293.objecter ms_dispatch 0x7fb358000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:02:02.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fb354ff9640 10 client.4293.objecter _finish_command 2 = system:0 2023-12-15T17:02:02.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fb354ff9640 20 client.4293.objecter put_session s=0x7fb344072860 osd=5 4 2023-12-15T17:02:02.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.591+0000 7fb354ff9640 15 client.4293.objecter _session_command_op_remove 5 2 2023-12-15T17:02:02.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.592+0000 7fb35e97c640 10 librados: watch_flush enter 2023-12-15T17:02:02.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.592+0000 7fb35e97c640 10 librados: watch_flush exit 2023-12-15T17:02:02.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.592+0000 7fb35e97c640 10 client.4293.objecter close_session for osd.5 2023-12-15T17:02:02.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.592+0000 7fb35e97c640 1 -- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] conn(0x7fb344072f30 msgr2=0x7fb344075350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.593+0000 7fb35e97c640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] conn(0x7fb344072f30 0x7fb344075350 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.593+0000 7fb35e97c640 20 client.4293.objecter put_session s=0x7fb344072860 osd=5 1 2023-12-15T17:02:02.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.593+0000 7fb35e97c640 20 client.4293.objecter shutdown clearing up homeless session... 2023-12-15T17:02:02.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.593+0000 7fb35e97c640 10 client.4293.objecter successfully canceled tick 2023-12-15T17:02:02.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.600+0000 7fb35e97c640 1 -- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fb34406cb50 msgr2=0x7fb34406f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.600+0000 7fb35e97c640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fb34406cb50 0x7fb34406f010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb3580abe10 tx=0x7fb34c007990 comp rx=0 tx=0).stop 2023-12-15T17:02:02.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.600+0000 7fb35e97c640 10 monclient: shutdown 2023-12-15T17:02:02.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.600+0000 7fb35e97c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.600+0000 7fb35e97c640 1 -- 172.21.15.69:0/484565075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3580a8a30 msgr2=0x7fb3580ac140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.600+0000 7fb35e97c640 1 --2- 172.21.15.69:0/484565075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3580a8a30 0x7fb3580ac140 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c7f0 tx=0x7fb34800ccb0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.600+0000 7fb35cf19640 1 -- 172.21.15.69:0/484565075 reap_dead start 2023-12-15T17:02:02.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.600+0000 7fb35e97c640 1 -- 172.21.15.69:0/484565075 shutdown_connections 2023-12-15T17:02:02.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.600+0000 7fb35e97c640 1 -- 172.21.15.69:0/484565075 >> 172.21.15.69:0/484565075 conn(0x7fb358061a30 msgr2=0x7fb3581458f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.600+0000 7fb35e97c640 1 -- 172.21.15.69:0/484565075 shutdown_connections 2023-12-15T17:02:02.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.600+0000 7fb35e97c640 1 -- 172.21.15.69:0/484565075 wait complete. 2023-12-15T17:02:02.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.601+0000 7fb35e97c640 1 librados: shutdown 2023-12-15T17:02:02.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.602+0000 7f80fb59e640 10 client.4296.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-15T17:02:02.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.602+0000 7f80fb59e640 15 client.4296.objecter _session_command_op_assign -1 2 2023-12-15T17:02:02.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.602+0000 7f80fb59e640 20 client.4296.objecter _get_session s=0x7f80dc0728e0 osd=2 3 2023-12-15T17:02:02.604 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.602+0000 7f80fb59e640 20 client.4296.objecter put_session s=0x7f80dc0728e0 osd=2 3 2023-12-15T17:02:02.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.602+0000 7f80fb59e640 20 client.4296.objecter _get_session s=0x7f80dc0728e0 osd=2 3 2023-12-15T17:02:02.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.602+0000 7f80fb59e640 15 client.4296.objecter _session_command_op_remove -1 2 2023-12-15T17:02:02.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.602+0000 7f80fb59e640 20 client.4296.objecter get_session s=0x7f80dc0728e0 osd=2 3 2023-12-15T17:02:02.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.602+0000 7f80fb59e640 15 client.4296.objecter _session_command_op_assign 2 2 2023-12-15T17:02:02.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.602+0000 7f80fb59e640 20 client.4296.objecter put_session s=0x7f80dc0728e0 osd=2 4 2023-12-15T17:02:02.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.602+0000 7f80fb59e640 10 client.4296.objecter _send_command 2 2023-12-15T17:02:02.605 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.602+0000 7f80fb59e640 1 -- 172.21.15.69:0/870852848 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f80f40366a0 con 0x7f80dc072fb0 2023-12-15T17:02:02.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdbb1ffb640 1 -- 172.21.15.69:0/2134763422 <== osd.3 v2:172.21.15.69:6824/882900488 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdbac009200 con 0x7fdba0072f30 2023-12-15T17:02:02.610 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdbb1ffb640 10 client.4286.objecter ms_dispatch 0x7fdbbc000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:02:02.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdbb1ffb640 10 client.4286.objecter _finish_command 2 = system:0 2023-12-15T17:02:02.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdbb1ffb640 20 client.4286.objecter put_session s=0x7fdba0072860 osd=3 4 2023-12-15T17:02:02.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdbb1ffb640 15 client.4286.objecter _session_command_op_remove 3 2 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdb9f7fe640 10 librados: watch_flush enter 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdb9f7fe640 10 librados: watch_flush exit 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdb9f7fe640 10 client.4286.objecter close_session for osd.3 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdb9f7fe640 1 -- 172.21.15.69:0/2134763422 >> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] conn(0x7fdba0072f30 msgr2=0x7fdba0075350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdb9f7fe640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] conn(0x7fdba0072f30 0x7fdba0075350 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdb9f7fe640 20 client.4286.objecter put_session s=0x7fdba0072860 osd=3 1 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdb9f7fe640 20 client.4286.objecter shutdown clearing up homeless session... 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.609+0000 7fdb9f7fe640 10 client.4286.objecter successfully canceled tick 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.610+0000 7fdb9f7fe640 1 -- 172.21.15.69:0/2134763422 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fdba006cb50 msgr2=0x7fdba006f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.610+0000 7fdb9f7fe640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fdba006cb50 0x7fdba006f010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdbb802b800 tx=0x7fdbb8030000 comp rx=0 tx=0).stop 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.610+0000 7fdb9f7fe640 10 monclient: shutdown 2023-12-15T17:02:02.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.610+0000 7fdb9f7fe640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.610+0000 7fdb9f7fe640 1 -- 172.21.15.69:0/2134763422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc1468b0 msgr2=0x7fdbbc141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.610+0000 7fdb9f7fe640 1 --2- 172.21.15.69:0/2134763422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbbc1468b0 0x7fdbbc141510 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400c6d0 tx=0x7fdbb400cb90 comp rx=0 tx=0).stop 2023-12-15T17:02:02.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.610+0000 7fdbc18ac640 1 -- 172.21.15.69:0/2134763422 reap_dead start 2023-12-15T17:02:02.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.610+0000 7fdb9f7fe640 1 -- 172.21.15.69:0/2134763422 shutdown_connections 2023-12-15T17:02:02.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.610+0000 7fdb9f7fe640 1 -- 172.21.15.69:0/2134763422 >> 172.21.15.69:0/2134763422 conn(0x7fdbbc0618d0 msgr2=0x7fdbbc070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.611+0000 7fdb9f7fe640 1 -- 172.21.15.69:0/2134763422 shutdown_connections 2023-12-15T17:02:02.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.611+0000 7fdb9f7fe640 1 -- 172.21.15.69:0/2134763422 wait complete. 2023-12-15T17:02:02.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.611+0000 7fdb9f7fe640 1 librados: shutdown 2023-12-15T17:02:02.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.612+0000 7f4594ff9640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-15T17:02:02.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.612+0000 7f4594ff9640 15 client.4318.objecter _session_command_op_assign -1 2 2023-12-15T17:02:02.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.612+0000 7f4594ff9640 20 client.4318.objecter _get_session s=0x7f45840728b0 osd=1 3 2023-12-15T17:02:02.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.612+0000 7f4594ff9640 20 client.4318.objecter put_session s=0x7f45840728b0 osd=1 3 2023-12-15T17:02:02.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.612+0000 7f4594ff9640 20 client.4318.objecter _get_session s=0x7f45840728b0 osd=1 3 2023-12-15T17:02:02.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.613+0000 7f4594ff9640 15 client.4318.objecter _session_command_op_remove -1 2 2023-12-15T17:02:02.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.613+0000 7f4594ff9640 20 client.4318.objecter get_session s=0x7f45840728b0 osd=1 3 2023-12-15T17:02:02.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.613+0000 7f4594ff9640 15 client.4318.objecter _session_command_op_assign 1 2 2023-12-15T17:02:02.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.613+0000 7f4594ff9640 20 client.4318.objecter put_session s=0x7f45840728b0 osd=1 4 2023-12-15T17:02:02.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.613+0000 7f4594ff9640 10 client.4318.objecter _send_command 2 2023-12-15T17:02:02.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.613+0000 7f4594ff9640 1 -- 172.21.15.69:0/2988014089 --> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f457c0011c0 con 0x7f4584072f80 2023-12-15T17:02:02.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80eaffd640 1 -- 172.21.15.69:0/870852848 <== osd.2 v2:172.21.15.69:6800/3322029776 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f80fc04f730 con 0x7f80dc072fb0 2023-12-15T17:02:02.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80eaffd640 10 client.4296.objecter ms_dispatch 0x7f80fc000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:02:02.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80eaffd640 10 client.4296.objecter _finish_command 2 = system:0 2023-12-15T17:02:02.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80eaffd640 20 client.4296.objecter put_session s=0x7f80dc0728e0 osd=2 4 2023-12-15T17:02:02.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80eaffd640 15 client.4296.objecter _session_command_op_remove 2 2 2023-12-15T17:02:02.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80e8ff9640 10 librados: watch_flush enter 2023-12-15T17:02:02.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80e8ff9640 10 librados: watch_flush exit 2023-12-15T17:02:02.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80e8ff9640 10 client.4296.objecter close_session for osd.2 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80e8ff9640 1 -- 172.21.15.69:0/870852848 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x7f80dc072fb0 msgr2=0x7f80dc0753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80e8ff9640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x7f80dc072fb0 0x7f80dc0753d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80e8ff9640 20 client.4296.objecter put_session s=0x7f80dc0728e0 osd=2 1 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80e8ff9640 20 client.4296.objecter shutdown clearing up homeless session... 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.614+0000 7f80e8ff9640 10 client.4296.objecter successfully canceled tick 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.615+0000 7f80e8ff9640 1 -- 172.21.15.69:0/870852848 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f80dc06cbd0 msgr2=0x7f80dc06f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.615+0000 7f80e8ff9640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f80dc06cbd0 0x7f80dc06f090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f80ec035000 tx=0x7f80ec030000 comp rx=0 tx=0).stop 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.615+0000 7f80e8ff9640 10 monclient: shutdown 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.615+0000 7f80e8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.615+0000 7f80e8ff9640 1 -- 172.21.15.69:0/870852848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f40c5770 msgr2=0x7f80f41605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.615+0000 7f80e8ff9640 1 --2- 172.21.15.69:0/870852848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f40c5770 0x7f80f41605a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f80f000e8d0 tx=0x7f80f000ed90 comp rx=0 tx=0).stop 2023-12-15T17:02:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.615+0000 7f80fad9d640 1 -- 172.21.15.69:0/870852848 reap_dead start 2023-12-15T17:02:02.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.616+0000 7f80e8ff9640 1 -- 172.21.15.69:0/870852848 shutdown_connections 2023-12-15T17:02:02.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.616+0000 7f80e8ff9640 1 -- 172.21.15.69:0/870852848 >> 172.21.15.69:0/870852848 conn(0x7f80f40065e0 msgr2=0x7f80f40c4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.617+0000 7f80e8ff9640 1 -- 172.21.15.69:0/870852848 shutdown_connections 2023-12-15T17:02:02.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.617+0000 7f80e8ff9640 1 -- 172.21.15.69:0/870852848 wait complete. 2023-12-15T17:02:02.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.617+0000 7f80e8ff9640 1 librados: shutdown 2023-12-15T17:02:02.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.619+0000 7f5ef4c09640 10 client.4299.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-15T17:02:02.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.619+0000 7f5ef4c09640 15 client.4299.objecter _session_command_op_assign -1 2 2023-12-15T17:02:02.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.619+0000 7f5ef4c09640 20 client.4299.objecter _get_session s=0x7f5ed0072860 osd=6 3 2023-12-15T17:02:02.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.619+0000 7f5ef4c09640 20 client.4299.objecter put_session s=0x7f5ed0072860 osd=6 3 2023-12-15T17:02:02.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.619+0000 7f5ef4c09640 20 client.4299.objecter _get_session s=0x7f5ed0072860 osd=6 3 2023-12-15T17:02:02.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.619+0000 7f5ef4c09640 15 client.4299.objecter _session_command_op_remove -1 2 2023-12-15T17:02:02.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.619+0000 7f5ef4c09640 20 client.4299.objecter get_session s=0x7f5ed0072860 osd=6 3 2023-12-15T17:02:02.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.619+0000 7f5ef4c09640 15 client.4299.objecter _session_command_op_assign 6 2 2023-12-15T17:02:02.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.619+0000 7f5ef4c09640 20 client.4299.objecter put_session s=0x7f5ed0072860 osd=6 4 2023-12-15T17:02:02.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.619+0000 7f5ef4c09640 10 client.4299.objecter _send_command 2 2023-12-15T17:02:02.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.619+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2563722376 --> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5ef0067ac0 con 0x7f5ed0072f30 2023-12-15T17:02:02.623 INFO:teuthology.orchestra.run.smithi069.stdout:60129542151 2023-12-15T17:02:02.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-15T17:02:02.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.625+0000 7f5edeffd640 1 -- 172.21.15.69:0/2563722376 <== osd.6 v2:172.21.15.192:6818/2563835645 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5ef0067ac0 con 0x7f5ed0072f30 2023-12-15T17:02:02.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.625+0000 7f5edeffd640 10 client.4299.objecter ms_dispatch 0x7f5ef0000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:02:02.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.625+0000 7f5edeffd640 10 client.4299.objecter _finish_command 2 = system:0 2023-12-15T17:02:02.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.625+0000 7f5edeffd640 20 client.4299.objecter put_session s=0x7f5ed0072860 osd=6 4 2023-12-15T17:02:02.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.625+0000 7f5edeffd640 15 client.4299.objecter _session_command_op_remove 6 2 2023-12-15T17:02:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.625+0000 7f5ef4c09640 10 librados: watch_flush enter 2023-12-15T17:02:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 10 librados: watch_flush exit 2023-12-15T17:02:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 10 client.4299.objecter close_session for osd.6 2023-12-15T17:02:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] conn(0x7f5ed0072f30 msgr2=0x7f5ed0075350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] conn(0x7f5ed0072f30 0x7f5ed0075350 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 20 client.4299.objecter put_session s=0x7f5ed0072860 osd=6 1 2023-12-15T17:02:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 20 client.4299.objecter shutdown clearing up homeless session... 2023-12-15T17:02:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 10 client.4299.objecter successfully canceled tick 2023-12-15T17:02:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f5ed006cb50 msgr2=0x7f5ed006f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f5ed006cb50 0x7f5ed006f010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0036000 tx=0x7f5ee0008030 comp rx=0 tx=0).stop 2023-12-15T17:02:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 10 monclient: shutdown 2023-12-15T17:02:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2563722376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef00a8e10 msgr2=0x7f5ef00b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 1 --2- 172.21.15.69:0/2563722376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef00a8e10 0x7f5ef00b5db0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400c700 tx=0x7f5ee400cb00 comp rx=0 tx=0).stop 2023-12-15T17:02:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5eeed9d640 1 -- 172.21.15.69:0/2563722376 reap_dead start 2023-12-15T17:02:02.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2563722376 shutdown_connections 2023-12-15T17:02:02.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.626+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2563722376 >> 172.21.15.69:0/2563722376 conn(0x7f5ef0061a30 msgr2=0x7f5ef00b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.628+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2563722376 shutdown_connections 2023-12-15T17:02:02.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.628+0000 7f5ef4c09640 1 -- 172.21.15.69:0/2563722376 wait complete. 2023-12-15T17:02:02.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.628+0000 7f4596ffd640 1 -- 172.21.15.69:0/2988014089 <== osd.1 v2:172.21.15.69:6808/1683202761 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f457c0011c0 con 0x7f4584072f80 2023-12-15T17:02:02.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.628+0000 7f4596ffd640 10 client.4318.objecter ms_dispatch 0x7f45a0000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:02:02.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.628+0000 7f4596ffd640 10 client.4318.objecter _finish_command 2 = system:0 2023-12-15T17:02:02.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.628+0000 7f4596ffd640 20 client.4318.objecter put_session s=0x7f45840728b0 osd=1 4 2023-12-15T17:02:02.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.628+0000 7f4596ffd640 15 client.4318.objecter _session_command_op_remove 1 2 2023-12-15T17:02:02.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.628+0000 7f5ef4c09640 1 librados: shutdown 2023-12-15T17:02:02.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.631+0000 7f4594ff9640 10 librados: watch_flush enter 2023-12-15T17:02:02.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.631+0000 7f4594ff9640 10 librados: watch_flush exit 2023-12-15T17:02:02.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.631+0000 7f4594ff9640 10 client.4318.objecter close_session for osd.1 2023-12-15T17:02:02.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.631+0000 7f4594ff9640 1 -- 172.21.15.69:0/2988014089 >> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] conn(0x7f4584072f80 msgr2=0x7f45840753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.631+0000 7f4594ff9640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] conn(0x7f4584072f80 0x7f45840753a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.631+0000 7f4594ff9640 20 client.4318.objecter put_session s=0x7f45840728b0 osd=1 1 2023-12-15T17:02:02.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.631+0000 7f4594ff9640 20 client.4318.objecter shutdown clearing up homeless session... 2023-12-15T17:02:02.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.631+0000 7f4594ff9640 10 client.4318.objecter successfully canceled tick 2023-12-15T17:02:02.634 INFO:teuthology.orchestra.run.smithi069.stdout:64424509445 2023-12-15T17:02:02.635 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-15T17:02:02.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.632+0000 7f4594ff9640 1 -- 172.21.15.69:0/2988014089 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f458406cba0 msgr2=0x7f458406f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.632+0000 7f4594ff9640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f458406cba0 0x7f458406f060 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4590000f60 tx=0x7f45900079b0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.632+0000 7f4594ff9640 10 monclient: shutdown 2023-12-15T17:02:02.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.632+0000 7f4594ff9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.632+0000 7f4594ff9640 1 -- 172.21.15.69:0/2988014089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45a00af360 msgr2=0x7f45a013f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.632+0000 7f4594ff9640 1 --2- 172.21.15.69:0/2988014089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45a00af360 0x7f45a013f240 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f459c00c9c0 tx=0x7f459c00ce80 comp rx=0 tx=0).stop 2023-12-15T17:02:02.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.633+0000 7f45a6aa2640 1 -- 172.21.15.69:0/2988014089 reap_dead start 2023-12-15T17:02:02.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.633+0000 7f4594ff9640 1 -- 172.21.15.69:0/2988014089 shutdown_connections 2023-12-15T17:02:02.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.633+0000 7f4594ff9640 1 -- 172.21.15.69:0/2988014089 >> 172.21.15.69:0/2988014089 conn(0x7f45a00b01e0 msgr2=0x7f45a0071b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.637 INFO:teuthology.orchestra.run.smithi069.stdout:64424509445 2023-12-15T17:02:02.637 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-15T17:02:02.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.633+0000 7f4594ff9640 1 -- 172.21.15.69:0/2988014089 shutdown_connections 2023-12-15T17:02:02.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.633+0000 7f4594ff9640 1 -- 172.21.15.69:0/2988014089 wait complete. 2023-12-15T17:02:02.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.633+0000 7f4594ff9640 1 librados: shutdown 2023-12-15T17:02:02.651 INFO:teuthology.orchestra.run.smithi069.stdout:60129542150 2023-12-15T17:02:02.652 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-15T17:02:02.657 INFO:teuthology.orchestra.run.smithi069.stdout:64424509447 2023-12-15T17:02:02.657 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-15T17:02:02.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.718+0000 7f89e332c640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.718+0000 7f89e332c640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.718+0000 7f89e332c640 10 monclient: monmap: 2023-12-15T17:02:02.720 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.720 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.721 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.719809+0000 2023-12-15T17:02:02.721 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.719809+0000 2023-12-15T17:02:02.721 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.721 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.721 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.721 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.721 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.721 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.721 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.721 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.719+0000 7f89e332c640 1 Processor -- start 2023-12-15T17:02:02.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.719+0000 7f89e332c640 1 -- start start 2023-12-15T17:02:02.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.719+0000 7f89e332c640 10 monclient: init 2023-12-15T17:02:02.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-12-15T17:02:02.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d402de00 0x7f89d402e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 10 monclient(hunting): picked mon.noname-b con 0x7f89d402de00 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89d402ea70 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89d40c48c0 0x7f89d40ce620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 10 monclient(hunting): picked mon.noname-a con 0x7f89d40c48c0 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d40c72f0 con 0x7f89d402ea70 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d4034e60 con 0x7f89d40c48c0 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d402e1e0 con 0x7f89d402de00 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f89e332c640 10 monclient(hunting): authenticate will time out at 1427.036976s 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.722+0000 7f89e2b2b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89d40c48c0 0x7f89d40ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.723+0000 7f89e2b2b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89d40c48c0 0x7f89d40ce620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58594/0 (socket says 172.21.15.69:58594) 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.723+0000 7f89e2b2b640 1 -- 172.21.15.69:0/914668095 learned_addr learned my addr 172.21.15.69:0/914668095 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.723+0000 7f89e2b2b640 10 monclient(hunting): get_auth_request con 0x7f89d40c48c0 auth_method 0 2023-12-15T17:02:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.723+0000 7f89e2b2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.723+0000 7f89e2b2b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.723+0000 7f89e2b2b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.719+0000 7f6ef6c7c640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.719+0000 7f6ef6c7c640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.719+0000 7f6ef6c7c640 10 monclient: monmap: 2023-12-15T17:02:02.731 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.731 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.731 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.720553+0000 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.720553+0000 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f6ef6c7c640 1 Processor -- start 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f6ef6c7c640 1 -- start start 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.720+0000 7f6ef6c7c640 10 monclient: init 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef00af360 0x7f6ef0071630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 10 monclient(hunting): picked mon.noname-b con 0x7f6ef00af360 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ef0071b70 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ef006fbc0 0x7f6ef006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 10 monclient(hunting): picked mon.noname-c con 0x7f6ef006fbc0 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef006f730 con 0x7f6ef0071b70 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ef00af740 con 0x7f6ef006fbc0 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef0070970 con 0x7f6ef00af360 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.721+0000 7f6ef6c7c640 10 monclient(hunting): authenticate will time out at 1427.037641s 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.722+0000 7f6eeffff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f6ef0071b70 0x7f6ef006e090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:60554/0 (socket says 172.21.15.69:60554) 2023-12-15T17:02:02.733 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.722+0000 7f6eeffff640 1 -- 172.21.15.69:0/231464937 learned_addr learned my addr 172.21.15.69:0/231464937 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.722+0000 7f6ef5219640 1 --2- 172.21.15.69:0/231464937 >> v2:172.21.15.192:3300/0 conn(0x7f6ef006fbc0 0x7f6ef006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.723+0000 7f6ef5219640 10 monclient(hunting): get_auth_request con 0x7f6ef006fbc0 auth_method 0 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.723+0000 7f6ef5219640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.725+0000 7f6ef5219640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.725+0000 7f6ef5219640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.723+0000 7f6ef4a18640 1 --2- 172.21.15.69:0/231464937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef00af360 0x7f6ef0071630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.726+0000 7f6ef4a18640 10 monclient(hunting): get_auth_request con 0x7f6ef00af360 auth_method 0 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.726+0000 7f6ef4a18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.726+0000 7f6ef4a18640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.726+0000 7f6ef4a18640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.729+0000 7f6eef7fe640 1 -- 172.21.15.69:0/231464937 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557040873 0 0) 0x7f6ef006f730 con 0x7f6ef0071b70 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.729+0000 7f6eef7fe640 10 monclient(hunting): _negotiate 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.729+0000 7f6eef7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.734 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.729+0000 7f6eef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.729+0000 7f6eef7fe640 10 monclient(hunting): my global_id is 4330 2023-12-15T17:02:02.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.729+0000 7f6eef7fe640 1 -- 172.21.15.69:0/231464937 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed8007930 con 0x7f6ef0071b70 2023-12-15T17:02:02.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.729+0000 7f6ef5219640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.729+0000 7f6ef5219640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f6ef5219640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.736 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f6ef4a18640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.737 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f6ef4a18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.737 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f6ef4a18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.737 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.725+0000 7f89e2b2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.737 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.734+0000 7f6ef5219640 10 monclient(hunting): handle_auth_done global_id 4333 payload 274 2023-12-15T17:02:02.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.734+0000 7f6ef5219640 1 -- 172.21.15.69:0/231464937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef00af360 msgr2=0x7f6ef0071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.734+0000 7f6ef5219640 1 --2- 172.21.15.69:0/231464937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef00af360 0x7f6ef0071630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.734+0000 7f6ef5219640 1 -- 172.21.15.69:0/231464937 >> v1:172.21.15.192:6789/0 conn(0x7f6ef0071b70 legacy=0x7f6ef006e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.734+0000 7f6ef5219640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.734+0000 7f6ef5219640 1 monclient: found mon.noname-c 2023-12-15T17:02:02.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.734+0000 7f6ef5219640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.734+0000 7f6ef5219640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.734+0000 7f6ef5219640 1 -- 172.21.15.69:0/231464937 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef0070c50 con 0x7f6ef006fbc0 2023-12-15T17:02:02.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.725+0000 7f89e2b2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.742 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.725+0000 7f89e2b2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.742 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.726+0000 7f89e232a640 1 --2- 172.21.15.69:0/914668095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d402de00 0x7f89d402e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.742 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.728+0000 7f89e232a640 10 monclient(hunting): get_auth_request con 0x7f89d402de00 auth_method 0 2023-12-15T17:02:02.742 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.728+0000 7f89e232a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.742 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.728+0000 7f89e232a640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.742 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.728+0000 7f89e232a640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.742 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.729+0000 7f89e232a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.729+0000 7f89e232a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.729+0000 7f89e232a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f89e2b2b640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f89e2b2b640 1 -- 172.21.15.69:0/914668095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d402de00 msgr2=0x7f89d402e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f89e2b2b640 1 --2- 172.21.15.69:0/914668095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d402de00 0x7f89d402e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f89e2b2b640 1 -- 172.21.15.69:0/914668095 >> v1:172.21.15.69:6789/0 conn(0x7f89d402ea70 legacy=0x7f89d40c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f89e2b2b640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f89e2b2b640 1 monclient: found mon.noname-a 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f89e2b2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f89e2b2b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.730+0000 7f89e2b2b640 1 -- 172.21.15.69:0/914668095 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d40c4d30 con 0x7f89d40c48c0 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.737+0000 7f89e2b2b640 1 --2- 172.21.15.69:0/914668095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d40c48c0 0x7f89d40ce620 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009440 tx=0x7f89d8034490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42661bb771b3f29a server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.743 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.740+0000 7f6ef5219640 1 --2- 172.21.15.69:0/231464937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ef006fbc0 0x7f6ef006e7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0008820 tx=0x7f6ee0008180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2feb970a6864bcfd server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f6eef7fe640 1 -- 172.21.15.69:0/231464937 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee002f830 con 0x7f6ef006fbc0 2023-12-15T17:02:02.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f6eef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f6eef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f6eef7fe640 10 monclient: dump: 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f6eef7fe640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f6eef7fe640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f6eef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f6eef7fe640 1 -- 172.21.15.69:0/231464937 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee002f990 con 0x7f6ef006fbc0 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f6eef7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f6eef7fe640 1 -- 172.21.15.69:0/231464937 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee002fd60 con 0x7f6ef006fbc0 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f6eef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f6eef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f6eef7fe640 10 monclient: dump: 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.748 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e1328640 1 -- 172.21.15.69:0/914668095 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d803b600 con 0x7f89d40c48c0 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e1328640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e1328640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e1328640 10 monclient: dump: 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.751 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e1328640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e1328640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e1328640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e332c640 5 monclient: authenticate success, global_id 4302 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e332c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e1328640 1 -- 172.21.15.69:0/914668095 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d8037050 con 0x7f89d40c48c0 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e1328640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e332c640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.744+0000 7f89e332c640 4 set_mon_vals no callback set 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f89e332c640 10 monclient: shutdown 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f89e332c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f89e332c640 1 -- 172.21.15.69:0/914668095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d40c48c0 msgr2=0x7f89d40ce620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.745+0000 7f89e332c640 1 --2- 172.21.15.69:0/914668095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d40c48c0 0x7f89d40ce620 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009440 tx=0x7f89d8034490 comp rx=0 tx=0).stop 2023-12-15T17:02:02.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.746+0000 7f89e332c640 1 -- 172.21.15.69:0/914668095 shutdown_connections 2023-12-15T17:02:02.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.746+0000 7f89e332c640 1 --2- 172.21.15.69:0/914668095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d40c48c0 0x7f89d40ce620 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.746+0000 7f89e332c640 1 --2- 172.21.15.69:0/914668095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d402de00 0x7f89d402e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.753 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.746+0000 7f89e332c640 1 -- 172.21.15.69:0/914668095 >> 172.21.15.69:0/914668095 conn(0x7f89d4005eb0 msgr2=0x7f89d40062c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.746+0000 7f6ef6c7c640 5 monclient: authenticate success, global_id 4333 2023-12-15T17:02:02.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.746+0000 7f6ef6c7c640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.746+0000 7f6ef6c7c640 4 set_mon_vals no callback set 2023-12-15T17:02:02.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.746+0000 7f6ef6c7c640 10 monclient: shutdown 2023-12-15T17:02:02.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.746+0000 7f6ef6c7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.746+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/231464937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ef006fbc0 msgr2=0x7f6ef006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.746+0000 7f6ef6c7c640 1 --2- 172.21.15.69:0/231464937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ef006fbc0 0x7f6ef006e7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0008820 tx=0x7f6ee0008180 comp rx=0 tx=0).stop 2023-12-15T17:02:02.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.748+0000 7fba3d890640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.754 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.748+0000 7fba3d890640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.748+0000 7fba3d890640 10 monclient: monmap: 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.749622+0000 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.749622+0000 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.756 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.748+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/231464937 shutdown_connections 2023-12-15T17:02:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.748+0000 7f6ef6c7c640 1 --2- 172.21.15.69:0/231464937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ef006fbc0 0x7f6ef006e7a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.748+0000 7f6ef6c7c640 1 --2- 172.21.15.69:0/231464937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef00af360 0x7f6ef0071630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.748+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/231464937 >> 172.21.15.69:0/231464937 conn(0x7f6ef00b01e0 msgr2=0x7f6ef00b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.748+0000 7f89e332c640 1 -- 172.21.15.69:0/914668095 shutdown_connections 2023-12-15T17:02:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.749+0000 7f89e332c640 1 -- 172.21.15.69:0/914668095 wait complete. 2023-12-15T17:02:02.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.749+0000 7fba3d890640 1 Processor -- start 2023-12-15T17:02:02.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.749+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/231464937 shutdown_connections 2023-12-15T17:02:02.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.749+0000 7fba3d890640 1 -- start start 2023-12-15T17:02:02.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.749+0000 7fba3d890640 10 monclient: init 2023-12-15T17:02:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.749+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/231464937 wait complete. 2023-12-15T17:02:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-12-15T17:02:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba30037580 0x7fba300c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 10 monclient(hunting): picked mon.noname-b con 0x7fba30037580 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fba300344f0 0x7fba300c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 10 monclient(hunting): picked mon.noname-a con 0x7fba300344f0 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba300c2260 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3002d5f0 con 0x7fba300c2260 2023-12-15T17:02:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba30037960 con 0x7fba300344f0 2023-12-15T17:02:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3002d8d0 con 0x7fba30037580 2023-12-15T17:02:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7fba3d890640 10 monclient(hunting): authenticate will time out at 1427.067261s 2023-12-15T17:02:02.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.749+0000 7f89e332c640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.749+0000 7f89e332c640 10 monclient: monmap: 2023-12-15T17:02:02.760 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.760 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.760 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.760 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.760 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.760 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.760 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.760 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.760 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.761 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7f89e332c640 1 librados: starting msgr at 2023-12-15T17:02:02.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7f89e332c640 1 librados: starting objecter 2023-12-15T17:02:02.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7f89e332c640 1 Processor -- start 2023-12-15T17:02:02.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7f89e332c640 1 -- start start 2023-12-15T17:02:02.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7f89e332c640 1 librados: setting wanted keys 2023-12-15T17:02:02.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7f89e332c640 1 librados: calling monclient init 2023-12-15T17:02:02.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7fba377fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fba300c2260 0x7fba300c5030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:60564/0 (socket says 172.21.15.69:60564) 2023-12-15T17:02:02.761 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7fba377fe640 1 -- 172.21.15.69:0/896063400 learned_addr learned my addr 172.21.15.69:0/896063400 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:02.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7f6ef6c7c640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7f6ef6c7c640 10 monclient: monmap: 2023-12-15T17:02:02.766 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.766 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.766 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.766 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.766 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.766 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.766 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f6ef6c7c640 1 librados: starting msgr at 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f6ef6c7c640 1 librados: starting objecter 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f6ef6c7c640 1 Processor -- start 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f6ef6c7c640 1 -- start start 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f6ef6c7c640 1 librados: setting wanted keys 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f6ef6c7c640 1 librados: calling monclient init 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f6ef6c7c640 10 monclient: init 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ef006fbc0 0x7f6ef0088b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 10 monclient(hunting): picked mon.noname-b con 0x7f6ef006fbc0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef0071b70 0x7f6ef01f4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 10 monclient(hunting): picked mon.noname-c con 0x7f6ef0071b70 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef00af360 0x7f6ef01f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 10 monclient(hunting): picked mon.noname-a con 0x7f6ef00af360 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef0087240 con 0x7f6ef00af360 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef01d9850 con 0x7f6ef006fbc0 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef00890d0 con 0x7f6ef0071b70 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f6ef6c7c640 10 monclient(hunting): authenticate will time out at 1427.069027s 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.753+0000 7f6eeffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef0071b70 0x7f6ef01f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.753+0000 7f6ef5219640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef00af360 0x7f6ef01f4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.753+0000 7f6eeffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef0071b70 0x7f6ef01f4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52930/0 (socket says 172.21.15.69:52930) 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6eeffff640 1 -- 172.21.15.69:0/1021920638 learned_addr learned my addr 172.21.15.69:0/1021920638 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6eeffff640 10 monclient(hunting): get_auth_request con 0x7f6ef0071b70 auth_method 0 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6eeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6eeffff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6eeffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6ef5219640 10 monclient(hunting): get_auth_request con 0x7f6ef00af360 auth_method 0 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6ef5219640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6ef5219640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6ef5219640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6eeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6eeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6eeffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6ef5219640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6ef5219640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.754+0000 7f6ef5219640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.755+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531428877 0 0) 0x7fba3002d5f0 con 0x7fba300c2260 2023-12-15T17:02:02.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.755+0000 7fba35ffb640 10 monclient(hunting): _negotiate 2023-12-15T17:02:02.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.755+0000 7fba35ffb640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.755+0000 7fba35ffb640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.755+0000 7fba35ffb640 10 monclient(hunting): my global_id is 4339 2023-12-15T17:02:02.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.755+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba20003520 con 0x7fba300c2260 2023-12-15T17:02:02.771 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f6ef4a18640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ef006fbc0 0x7f6ef0088b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.771 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f6ef4a18640 10 monclient(hunting): get_auth_request con 0x7f6ef006fbc0 auth_method 0 2023-12-15T17:02:02.771 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f6ef4a18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.771 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f6ef4a18640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.771 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f6ef4a18640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.750+0000 7f89e332c640 10 monclient: init 2023-12-15T17:02:02.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:02:02.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d402de00 0x7f89d4030f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 10 monclient(hunting): picked mon.noname-a con 0x7f89d402de00 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d402ea70 0x7f89d4031c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 10 monclient(hunting): picked mon.noname-b con 0x7f89d402ea70 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d40c48c0 0x7f89d4169f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 10 monclient(hunting): picked mon.noname-c con 0x7f89d40c48c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d40c0700 con 0x7f89d402de00 2023-12-15T17:02:02.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d4169a50 con 0x7f89d402ea70 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d40beb60 con 0x7f89d40c48c0 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.751+0000 7f89e332c640 10 monclient(hunting): authenticate will time out at 1427.067903s 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f89e2b2b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d40c48c0 0x7f89d4169f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.752+0000 7f89e1b29640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d402ea70 0x7f89d4031c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.753+0000 7f89e1b29640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d402ea70 0x7f89d4031c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37264/0 (socket says 172.21.15.69:37264) 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.753+0000 7f89e1b29640 1 -- 172.21.15.69:0/670731553 learned_addr learned my addr 172.21.15.69:0/670731553 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.753+0000 7f89e1b29640 10 monclient(hunting): get_auth_request con 0x7f89d402ea70 auth_method 0 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.753+0000 7f89e1b29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.753+0000 7f89e1b29640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.753+0000 7f89e1b29640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f89e1b29640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f89e1b29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f89e1b29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f89e2b2b640 10 monclient(hunting): get_auth_request con 0x7f89d40c48c0 auth_method 0 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f89e2b2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f89e2b2b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.756+0000 7f89e2b2b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.757+0000 7f89e232a640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d402de00 0x7f89d4030f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.757+0000 7f89e232a640 10 monclient(hunting): get_auth_request con 0x7f89d402de00 auth_method 0 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.757+0000 7f89e232a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.757+0000 7f89e232a640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.757+0000 7f89e232a640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f6eeffff640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2023-12-15T17:02:02.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f6eeffff640 1 -- 172.21.15.69:0/1021920638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ef006fbc0 msgr2=0x7f6ef0088b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f6eeffff640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ef006fbc0 0x7f6ef0088b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f6eeffff640 1 -- 172.21.15.69:0/1021920638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef00af360 msgr2=0x7f6ef01f4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f6eeffff640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef00af360 0x7f6ef01f4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f6eeffff640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f6eeffff640 1 monclient: found mon.noname-c 2023-12-15T17:02:02.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f6eeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f6eeffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.778 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f6eeffff640 1 -- 172.21.15.69:0/1021920638 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee0008480 con 0x7f6ef0071b70 2023-12-15T17:02:02.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f89e2b2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f89e2b2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.762+0000 7f89e2b2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.763+0000 7f89e1b29640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2023-12-15T17:02:02.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.763+0000 7f89e1b29640 1 -- 172.21.15.69:0/670731553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d40c48c0 msgr2=0x7f89d4169f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.763+0000 7f89e1b29640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d40c48c0 0x7f89d4169f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.763+0000 7f89e1b29640 1 -- 172.21.15.69:0/670731553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d402de00 msgr2=0x7f89d4030f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.763+0000 7f89e1b29640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d402de00 0x7f89d4030f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.763+0000 7f89e1b29640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.763+0000 7f89e1b29640 1 monclient: found mon.noname-b 2023-12-15T17:02:02.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.763+0000 7f89e1b29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.763+0000 7f89e1b29640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.763+0000 7f89e1b29640 1 -- 172.21.15.69:0/670731553 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d8009d60 con 0x7f89d402ea70 2023-12-15T17:02:02.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1471551508 0 0) 0x7fba20003520 con 0x7fba300c2260 2023-12-15T17:02:02.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba30037580 msgr2=0x7fba300c1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba30037580 0x7fba300c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 >> v2:172.21.15.69:3300/0 conn(0x7fba300344f0 msgr2=0x7fba300c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 1 --2- 172.21.15.69:0/896063400 >> v2:172.21.15.69:3300/0 conn(0x7fba300344f0 0x7fba300c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 1 monclient: found mon.noname-c-legacy 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.192:6789/0 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3002b5d0 con 0x7fba300c2260 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.764+0000 7fba35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba3d890640 5 monclient: authenticate success, global_id 4339 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba3d890640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2578675164 0 0) 0x7fba2c001670 con 0x7fba300c2260 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient: dump: 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] but i'm connected to v1:172.21.15.192:6789/0, reconnecting 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 >> v1:172.21.15.192:6789/0 conn(0x7fba300c2260 legacy=0x7fba300c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba20003970 0x7fba20003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient(hunting): picked mon.b con 0x7fba20003970 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba30037580 0x7fba200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient(hunting): picked mon.c con 0x7fba30037580 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba20004e20 0x7fba2000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient(hunting): picked mon.a con 0x7fba20004e20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba30037960 con 0x7fba20004e20 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3002d8d0 con 0x7fba20003970 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba20005660 con 0x7fba30037580 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba35ffb640 10 monclient(hunting): _finish_auth 0 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.765+0000 7fba3d890640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.769+0000 7fba36ffd640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba20003970 0x7fba20003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.769+0000 7fba36ffd640 10 monclient(hunting): get_auth_request con 0x7fba20003970 auth_method 0 2023-12-15T17:02:02.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.769+0000 7fba36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.769+0000 7fba36ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.769+0000 7fba36ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:02:02.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.769+0000 7f89e1b29640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d402ea70 0x7f89d4031c10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00c410 tx=0x7f89cc00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.770+0000 7f89d2ffd640 10 client.?.objecter ms_handle_connect 0x7f89d402ea70 2023-12-15T17:02:02.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.770+0000 7f89d2ffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.770+0000 7fba377fe640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba20004e20 0x7fba2000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.770+0000 7fba377fe640 10 monclient(hunting): get_auth_request con 0x7fba20004e20 auth_method 0 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.770+0000 7fba377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.770+0000 7fba377fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.770+0000 7fba377fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.771+0000 7fba36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.771+0000 7fba36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.771+0000 7fba36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.771+0000 7fba377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.771+0000 7fba377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.771+0000 7fba377fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba367fc640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba30037580 0x7fba200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 10 monclient(hunting): handle_auth_done global_id 4339 payload 293 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 1 -- 172.21.15.69:0/896063400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba30037580 msgr2=0x7fba200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba30037580 0x7fba200047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 1 -- 172.21.15.69:0/896063400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba20004e20 msgr2=0x7fba2000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba20004e20 0x7fba2000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 1 monclient: found mon.b 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 1 -- 172.21.15.69:0/896063400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20003520 con 0x7fba20003970 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba367fc640 10 monclient: get_auth_request con 0x7fba30037580 auth_method 0 2023-12-15T17:02:02.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7fba367fc640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba30037580 0x7fba200047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-15T17:02:02.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89d2ffd640 1 -- 172.21.15.69:0/670731553 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc0035f0 con 0x7f89d402ea70 2023-12-15T17:02:02.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:02.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89d2ffd640 10 monclient: dump: 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89d2ffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89d2ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89e332c640 5 monclient: authenticate success, global_id 4336 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89e332c640 10 monclient: _renew_subs 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89e332c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89e332c640 1 -- 172.21.15.69:0/670731553 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d4032150 con 0x7f89d402ea70 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.775+0000 7f89e332c640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.776+0000 7f89e332c640 10 monclient: _renew_subs 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.776+0000 7f89e332c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.776+0000 7f89e332c640 1 -- 172.21.15.69:0/670731553 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d416a4b0 con 0x7f89d402ea70 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.776+0000 7f89e332c640 1 librados: init done 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.776+0000 7f89d2ffd640 1 -- 172.21.15.69:0/670731553 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89cc003750 con 0x7f89d402ea70 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.776+0000 7f89d2ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.776+0000 7f89e0b27640 4 set_mon_vals no callback set 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.777+0000 7f89d2ffd640 1 -- 172.21.15.69:0/670731553 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc003b40 con 0x7f89d402ea70 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.777+0000 7f89d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.777+0000 7f89d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.777+0000 7f89d2ffd640 10 monclient: dump: 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.780+0000 7f89e332c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.780+0000 7f89e332c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.780+0000 7f89e332c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:02.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.780+0000 7f89e332c640 1 -- 172.21.15.69:0/670731553 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c4005040 con 0x7f89d402ea70 2023-12-15T17:02:02.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.780+0000 7f6eeffff640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef0071b70 0x7f6ef01f4950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6ee800cbc0 tx=0x7f6ee800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.780+0000 7f6eed7fa640 10 client.?.objecter ms_handle_connect 0x7f6ef0071b70 2023-12-15T17:02:02.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.780+0000 7f6eed7fa640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:02.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7fba36ffd640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba20003970 0x7fba20003d90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba28009860 tx=0x7fba200057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=873f52677f7b1eb1 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 1 -- 172.21.15.69:0/1021920638 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee800ad40 con 0x7f6ef0071b70 2023-12-15T17:02:02.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:02.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 10 monclient: dump: 2023-12-15T17:02:02.793 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.793 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.793 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.793 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.793 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.793 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 1 -- 172.21.15.69:0/1021920638 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee8012400 con 0x7f6ef0071b70 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 1 -- 172.21.15.69:0/1021920638 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee800aea0 con 0x7f6ef0071b70 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eed7fa640 10 monclient: dump: 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.794 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6ef6c7c640 5 monclient: authenticate success, global_id 4295 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6ef6c7c640 10 monclient: _renew_subs 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6ef6c7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/1021920638 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef01f53d0 con 0x7f6ef0071b70 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6ef6c7c640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6ef6c7c640 10 monclient: _renew_subs 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6ef6c7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/1021920638 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef0087ff0 con 0x7f6ef0071b70 2023-12-15T17:02:02.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6ef6c7c640 1 librados: init done 2023-12-15T17:02:02.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.785+0000 7f6eeeffd640 4 set_mon_vals no callback set 2023-12-15T17:02:02.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.787+0000 7f6ef6c7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.787+0000 7f6ef6c7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.787+0000 7f6ef6c7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:02.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.787+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/1021920638 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed0005040 con 0x7f6ef0071b70 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.794+0000 7fc183fff640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.794+0000 7fc183fff640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.794+0000 7fc183fff640 10 monclient: monmap: 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.795755+0000 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.795755+0000 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.797 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.798 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.798 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.795+0000 7fc183fff640 1 Processor -- start 2023-12-15T17:02:02.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.795+0000 7fc183fff640 1 -- start start 2023-12-15T17:02:02.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.795+0000 7fc183fff640 10 monclient: init 2023-12-15T17:02:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-12-15T17:02:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc17c038d80 0x7fc17c0c5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 10 monclient(hunting): picked mon.noname-b con 0x7fc17c038d80 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc17c039160 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc17c0c5770 0x7fc17c034430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc17c0c5770 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17c0c7e70 con 0x7fc17c039160 2023-12-15T17:02:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17c037610 con 0x7fc17c0c5770 2023-12-15T17:02:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c035870 con 0x7fc17c038d80 2023-12-15T17:02:02.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.797+0000 7fc183fff640 10 monclient(hunting): authenticate will time out at 1427.113814s 2023-12-15T17:02:02.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.799+0000 7fc1827fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc17c039160 0x7fc17c0c8320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59700/0 (socket says 172.21.15.69:59700) 2023-12-15T17:02:02.801 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.799+0000 7fc1827fc640 1 -- 172.21.15.69:0/3003911875 learned_addr learned my addr 172.21.15.69:0/3003911875 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:02.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.800+0000 7f89d2ffd640 1 -- 172.21.15.69:0/670731553 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f89cc003ca0 con 0x7f89d402ea70 2023-12-15T17:02:02.802 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.800+0000 7f89d2ffd640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f89c006cbd0 0x7f89c006f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.801+0000 7f89e232a640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f89c006cbd0 0x7f89c006f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.801+0000 7f89e232a640 10 monclient: get_auth_request con 0x7f89c006cbd0 auth_method 0 2023-12-15T17:02:02.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.801+0000 7f89e232a640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f89c006cbd0 0x7f89c006f090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f89dc068a20 tx=0x7f89dc06e040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.801+0000 7f89d2ffd640 10 client.4336.objecter ms_handle_connect 0x7f89c006cbd0 2023-12-15T17:02:02.805 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.803+0000 7f89d2ffd640 1 -- 172.21.15.69:0/670731553 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f89cc0010c0 con 0x7f89d402ea70 2023-12-15T17:02:02.805 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.803+0000 7f89d2ffd640 10 client.4336.objecter ms_dispatch 0x7f89dc000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:02.805 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.803+0000 7f89d2ffd640 3 client.4336.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:02.805 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.803+0000 7f89d2ffd640 3 client.4336.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:02.805 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.803+0000 7f89d2ffd640 20 client.4336.objecter dump_active .. 0 homeless 2023-12-15T17:02:02.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.804+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95700565 0 0) 0x7fc17c0c7e70 con 0x7fc17c039160 2023-12-15T17:02:02.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.804+0000 7fc181ffb640 10 monclient(hunting): _negotiate 2023-12-15T17:02:02.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.804+0000 7fc181ffb640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.804+0000 7fc181ffb640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.804+0000 7fc181ffb640 10 monclient(hunting): my global_id is 4311 2023-12-15T17:02:02.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.804+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc16c003520 con 0x7fc17c039160 2023-12-15T17:02:02.806 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.804+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28032600 con 0x7fba20003970 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.804+0000 7fba35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.804+0000 7fba35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.804+0000 7fba35ffb640 10 monclient: dump: 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.807 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.805+0000 7fba3d890640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2263773607 0 0) 0x7fc16c003520 con 0x7fc17c039160 2023-12-15T17:02:02.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc17c038d80 msgr2=0x7fc17c0c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 --2- 172.21.15.69:0/3003911875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc17c038d80 0x7fc17c0c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 >> v2:172.21.15.192:3300/0 conn(0x7fc17c0c5770 msgr2=0x7fc17c034430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 --2- 172.21.15.69:0/3003911875 >> v2:172.21.15.192:3300/0 conn(0x7fc17c0c5770 0x7fc17c034430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 monclient: found mon.noname-a-legacy 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.69:6789/0 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c036070 con 0x7fc17c039160 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2578675164 0 0) 0x7fc174001650 con 0x7fc17c039160 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient: dump: 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.811 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] but i'm connected to v1:172.21.15.69:6789/0, reconnecting 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 >> v1:172.21.15.69:6789/0 conn(0x7fc17c039160 legacy=0x7fc17c0c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 --2- 172.21.15.69:0/3003911875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc16c003990 0x7fc16c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient(hunting): picked mon.a con 0x7fc16c003990 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 --2- 172.21.15.69:0/3003911875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc16c0047c0 0x7fc16c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient(hunting): picked mon.b con 0x7fc16c0047c0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 --2- 172.21.15.69:0/3003911875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c005380 0x7fc16c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient(hunting): picked mon.c con 0x7fc16c005380 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c037610 con 0x7fc16c003990 2023-12-15T17:02:02.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c035870 con 0x7fc16c0047c0 2023-12-15T17:02:02.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c00c290 con 0x7fc16c005380 2023-12-15T17:02:02.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.813 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.808+0000 7fc181ffb640 10 monclient(hunting): _finish_auth 0 2023-12-15T17:02:02.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.812+0000 7f52bf644640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.812+0000 7f52bf644640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.812+0000 7f52bf644640 10 monclient: monmap: 2023-12-15T17:02:02.814 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.814 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.814 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.814051+0000 2023-12-15T17:02:02.814 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.814051+0000 2023-12-15T17:02:02.814 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.814 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.815 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.815 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.815 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.815 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.815 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.815 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.810+0000 7fc182ffd640 1 --2- 172.21.15.69:0/3003911875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc16c003990 0x7fc16c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.812+0000 7fc182ffd640 10 monclient(hunting): get_auth_request con 0x7fc16c003990 auth_method 0 2023-12-15T17:02:02.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.812+0000 7fc182ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.812+0000 7fc182ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.815 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.812+0000 7fc182ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:02:02.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.813+0000 7f52bf644640 1 Processor -- start 2023-12-15T17:02:02.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.813+0000 7f52bf644640 1 -- start start 2023-12-15T17:02:02.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.813+0000 7f52bf644640 10 monclient: init 2023-12-15T17:02:02.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-12-15T17:02:02.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52b80b0f20 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:02.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52b80b29e0 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:02.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f52b8143950 0x7f52b8151c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 10 monclient(hunting): picked mon.noname-a con 0x7f52b8143950 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:02.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b8143d80 con 0x7f52b80b0f20 2023-12-15T17:02:02.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b806dc60 con 0x7f52b80b29e0 2023-12-15T17:02:02.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b8146cf0 con 0x7f52b8143950 2023-12-15T17:02:02.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7f52bf644640 10 monclient(hunting): authenticate will time out at 1427.130906s 2023-12-15T17:02:02.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.814+0000 7fc182ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.815+0000 7fc182ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.815+0000 7fc182ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-15T17:02:02.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.815+0000 7f52bdbe1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f52b8143950 0x7f52b8151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.816+0000 7f692359e640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.816+0000 7f692359e640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.816+0000 7f692359e640 10 monclient: monmap: 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.817574+0000 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.817574+0000 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7f692359e640 1 Processor -- start 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7f692359e640 1 -- start start 2023-12-15T17:02:02.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7f692359e640 10 monclient: init 2023-12-15T17:02:02.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.818+0000 7f692359e640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.818+0000 7f692359e640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-12-15T17:02:02.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.818+0000 7f692359e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f691c038d80 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:02.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 10 monclient(hunting): handle_auth_done global_id 4311 payload 293 2023-12-15T17:02:02.820 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 1 -- 172.21.15.69:0/3003911875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c005380 msgr2=0x7fc16c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 1 --2- 172.21.15.69:0/3003911875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c005380 0x7fc16c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 1 -- 172.21.15.69:0/3003911875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc16c0047c0 msgr2=0x7fc16c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 1 --2- 172.21.15.69:0/3003911875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc16c0047c0 0x7fc16c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 1 monclient: found mon.a 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 1 -- 172.21.15.69:0/3003911875 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c003520 con 0x7fc16c003990 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.817+0000 7fc182ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.818+0000 7fc183fff640 5 monclient: authenticate success, global_id 4311 2023-12-15T17:02:02.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.818+0000 7fc183fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.818+0000 7f692359e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f691c039160 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:02.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.818+0000 7f692359e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f691c0c8a30 0x7f691c0d3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.818+0000 7f692359e640 10 monclient(hunting): picked mon.noname-c con 0x7f691c0c8a30 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:02.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.819+0000 7f692359e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c0c7e70 con 0x7f691c039160 2023-12-15T17:02:02.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.819+0000 7f692359e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c035440 con 0x7f691c038d80 2023-12-15T17:02:02.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.819+0000 7f692359e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.819+0000 7f692359e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691c037610 con 0x7f691c0c8a30 2023-12-15T17:02:02.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.819+0000 7f692359e640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.819+0000 7f692359e640 10 monclient(hunting): authenticate will time out at 1427.135543s 2023-12-15T17:02:02.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.820+0000 7f5437b5a640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.820+0000 7f5437b5a640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.820+0000 7f5437b5a640 10 monclient: monmap: 2023-12-15T17:02:02.823 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.821615+0000 2023-12-15T17:02:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.821615+0000 2023-12-15T17:02:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.824 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.824 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.820+0000 7f52bdbe1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f52b8143950 0x7f52b8151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58648/0 (socket says 172.21.15.69:58648) 2023-12-15T17:02:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.820+0000 7f52bdbe1640 1 -- 172.21.15.69:0/907165524 learned_addr learned my addr 172.21.15.69:0/907165524 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.820+0000 7f52bdbe1640 10 monclient(hunting): get_auth_request con 0x7f52b8143950 auth_method 0 2023-12-15T17:02:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.820+0000 7f52bdbe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.820+0000 7f52bdbe1640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.820+0000 7f52bdbe1640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.821+0000 7f5437b5a640 1 Processor -- start 2023-12-15T17:02:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.821+0000 7f5437b5a640 1 -- start start 2023-12-15T17:02:02.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.821+0000 7f5437b5a640 10 monclient: init 2023-12-15T17:02:02.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.821+0000 7f52bdbe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.821+0000 7f52bdbe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.821+0000 7f52bdbe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-12-15T17:02:02.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54300a8a30 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:02.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5430142bf0 0x7f543013ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 10 monclient(hunting): picked mon.noname-a con 0x7f5430142bf0 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54300a8e10 0x7f543013f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 10 monclient(hunting): picked mon.noname-b con 0x7f54300a8e10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54300aac10 con 0x7f54300a8a30 2023-12-15T17:02:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54300aaa10 con 0x7f5430142bf0 2023-12-15T17:02:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f6eed7fa640 1 -- 172.21.15.69:0/1021920638 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6ee8007070 con 0x7f6ef0071b70 2023-12-15T17:02:02.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f6eed7fa640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6edc06cba0 0x7f6edc06f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54300aaef0 con 0x7f54300a8e10 2023-12-15T17:02:02.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f5437b5a640 10 monclient(hunting): authenticate will time out at 1427.138615s 2023-12-15T17:02:02.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f6ef4a18640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6edc06cba0 0x7f6edc06f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.822+0000 7f6ef4a18640 10 monclient: get_auth_request con 0x7f6edc06cba0 auth_method 0 2023-12-15T17:02:02.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.823+0000 7fc182ffd640 1 --2- 172.21.15.69:0/3003911875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc16c003990 0x7fc16c003dd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0075a0 tx=0x7fc16c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d378c60dee45d374 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.823+0000 7f6ef4a18640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6edc06cba0 0x7f6edc06f060 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400bb00 tx=0x7f6ee4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.823+0000 7f6eed7fa640 10 client.4295.objecter ms_handle_connect 0x7f6edc06cba0 2023-12-15T17:02:02.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.825+0000 7f54350f5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5430142bf0 0x7f543013ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.827+0000 7f52bdbe1640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2023-12-15T17:02:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.827+0000 7f52bdbe1640 1 -- 172.21.15.69:0/907165524 >> v1:172.21.15.192:6789/0 conn(0x7f52b80b29e0 legacy=0x7f52b8143240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.827+0000 7f52bdbe1640 1 -- 172.21.15.69:0/907165524 >> v1:172.21.15.69:6789/0 conn(0x7f52b80b0f20 legacy=0x7f52b80b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.827+0000 7f52bdbe1640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.827+0000 7f52bdbe1640 1 monclient: found mon.noname-a 2023-12-15T17:02:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.827+0000 7f52bdbe1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.827+0000 7f52bdbe1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.827+0000 7f52bdbe1640 1 -- 172.21.15.69:0/907165524 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b806e800 con 0x7f52b8143950 2023-12-15T17:02:02.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.828+0000 7f54358f6640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f54300a8a30 0x7f54301424e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59734/0 (socket says 172.21.15.69:59734) 2023-12-15T17:02:02.832 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.828+0000 7f54358f6640 1 -- 172.21.15.69:0/1713454001 learned_addr learned my addr 172.21.15.69:0/1713454001 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:02.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.834+0000 7f52bdbe1640 1 --2- 172.21.15.69:0/907165524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52b8143950 0x7f52b8151c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009e50 tx=0x7f52b002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3428bd1becb53b6 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.838 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.836+0000 7f54348f4640 1 -- 172.21.15.69:0/1713454001 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124862260 0 0) 0x7f54300aac10 con 0x7f54300a8a30 2023-12-15T17:02:02.838 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.836+0000 7f54348f4640 10 monclient(hunting): _negotiate 2023-12-15T17:02:02.838 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.836+0000 7f54348f4640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.838 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.836+0000 7f54348f4640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.838 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.836+0000 7f54348f4640 10 monclient(hunting): my global_id is 4317 2023-12-15T17:02:02.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.836+0000 7f54348f4640 1 -- 172.21.15.69:0/1713454001 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f541c003520 con 0x7f54300a8a30 2023-12-15T17:02:02.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.837+0000 7f54350f5640 10 monclient(hunting): get_auth_request con 0x7f5430142bf0 auth_method 0 2023-12-15T17:02:02.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.837+0000 7f54350f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.837+0000 7f54350f5640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.839 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.837+0000 7f54350f5640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.837+0000 7f6921d9b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f691c039160 0x7f691c0c8320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59722/0 (socket says 172.21.15.69:59722) 2023-12-15T17:02:02.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.837+0000 7f6921d9b640 1 -- 172.21.15.69:0/467680622 learned_addr learned my addr 172.21.15.69:0/467680622 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:02.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.839+0000 7f54350f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.839+0000 7f54350f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.840 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.839+0000 7f54350f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.843+0000 7f54350f5640 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2023-12-15T17:02:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.843+0000 7f54350f5640 1 -- 172.21.15.69:0/1713454001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54300a8e10 msgr2=0x7f543013f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.843+0000 7f54350f5640 1 --2- 172.21.15.69:0/1713454001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54300a8e10 0x7f543013f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.843+0000 7f54350f5640 1 -- 172.21.15.69:0/1713454001 >> v1:172.21.15.69:6789/0 conn(0x7f54300a8a30 legacy=0x7f54301424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.843+0000 7f54350f5640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.843+0000 7f54350f5640 1 monclient: found mon.noname-a 2023-12-15T17:02:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.843+0000 7f54350f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.843+0000 7f54350f5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.843+0000 7f54350f5640 1 -- 172.21.15.69:0/1713454001 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54300a97f0 con 0x7f5430142bf0 2023-12-15T17:02:02.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.854+0000 7f54350f5640 1 --2- 172.21.15.69:0/1713454001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5430142bf0 0x7f543013ecf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f54200097a0 tx=0x7f542002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1274deb92706a8 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 1 -- 172.21.15.69:0/907165524 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b002ca80 con 0x7f52b8143950 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 10 monclient: dump: 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.858 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 1 -- 172.21.15.69:0/907165524 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b002cbe0 con 0x7f52b8143950 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 1 -- 172.21.15.69:0/907165524 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b00343f0 con 0x7f52b8143950 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52affff640 10 monclient: dump: 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.860 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52bf644640 5 monclient: authenticate success, global_id 4314 2023-12-15T17:02:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52bf644640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7f52bf644640 4 set_mon_vals no callback set 2023-12-15T17:02:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7febfff16640 10 monclient: get_monmap_and_config 2023-12-15T17:02:02.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7febfff16640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.856+0000 7febfff16640 10 monclient: monmap: 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:02.858296+0000 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:02.858296+0000 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f52bf644640 10 monclient: shutdown 2023-12-15T17:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f52bf644640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f52bf644640 1 -- 172.21.15.69:0/907165524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52b8143950 msgr2=0x7f52b8151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f52bf644640 1 --2- 172.21.15.69:0/907165524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52b8143950 0x7f52b8151c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009e50 tx=0x7f52b002bee0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528911398 0 0) 0x7f691c0c7e70 con 0x7f691c039160 2023-12-15T17:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f692159a640 10 monclient(hunting): _negotiate 2023-12-15T17:02:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f52bf644640 1 -- 172.21.15.69:0/907165524 shutdown_connections 2023-12-15T17:02:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f52bf644640 1 --2- 172.21.15.69:0/907165524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52b8143950 0x7f52b8151c90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f692159a640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f52bf644640 1 -- 172.21.15.69:0/907165524 >> 172.21.15.69:0/907165524 conn(0x7f52b8061a30 msgr2=0x7f52b80b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f692159a640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f692159a640 10 monclient(hunting): my global_id is 4323 2023-12-15T17:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f690c003520 con 0x7f691c039160 2023-12-15T17:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f52bf644640 1 -- 172.21.15.69:0/907165524 shutdown_connections 2023-12-15T17:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7febfff16640 1 Processor -- start 2023-12-15T17:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7f52bf644640 1 -- 172.21.15.69:0/907165524 wait complete. 2023-12-15T17:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7febfff16640 1 -- start start 2023-12-15T17:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.857+0000 7febfff16640 10 monclient: init 2023-12-15T17:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febf80a8a30 0x7febf81424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 10 monclient(hunting): picked mon.noname-b con 0x7febf80a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7febf80a8e10 0x7febf8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 10 monclient(hunting): picked mon.noname-c con 0x7febf80a8e10 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febf81454a0 0x7febf8145930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 10 monclient(hunting): picked mon.noname-a con 0x7febf81454a0 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf80aaa10 con 0x7febf81454a0 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf80aac10 con 0x7febf80a8e10 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf80aaef0 con 0x7febf80a8a30 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7febfff16640 10 monclient(hunting): authenticate will time out at 1427.174919s 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7f52bf644640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.858+0000 7f52bf644640 10 monclient: monmap: 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.859+0000 7f52bf644640 1 librados: starting msgr at 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.859+0000 7f52bf644640 1 librados: starting objecter 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.859+0000 7f52bf644640 1 Processor -- start 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.859+0000 7f52bf644640 1 -- start start 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.859+0000 7f52bf644640 1 librados: setting wanted keys 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.859+0000 7f52bf644640 1 librados: calling monclient init 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.859+0000 7f52bf644640 10 monclient: init 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52b80b0f20 0x7f52b8124330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 10 monclient(hunting): picked mon.noname-a con 0x7f52b80b0f20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52b80b29e0 0x7f52b813f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 10 monclient(hunting): picked mon.noname-b con 0x7f52b80b29e0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52b8143950 0x7f52b813fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 10 monclient(hunting): picked mon.noname-c con 0x7f52b8143950 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b81d8da0 con 0x7f52b80b0f20 2023-12-15T17:02:02.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b8122cb0 con 0x7f52b80b29e0 2023-12-15T17:02:02.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b8124bc0 con 0x7f52b8143950 2023-12-15T17:02:02.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.860+0000 7f52bf644640 10 monclient(hunting): authenticate will time out at 1427.176524s 2023-12-15T17:02:02.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178032600 con 0x7fc16c003990 2023-12-15T17:02:02.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc181ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc181ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:02.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc181ffb640 10 monclient: dump: 2023-12-15T17:02:02.871 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.871 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17802e050 con 0x7fc16c003990 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc183fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc181ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc183fff640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc181ffb640 1 -- 172.21.15.69:0/3003911875 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1780329c0 con 0x7fc16c003990 2023-12-15T17:02:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc181ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc183fff640 4 set_mon_vals no callback set 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc181ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.862+0000 7fc181ffb640 10 monclient: dump: 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.873 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.863+0000 7fc183fff640 10 monclient: shutdown 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.863+0000 7fc183fff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.863+0000 7fc183fff640 1 -- 172.21.15.69:0/3003911875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc16c003990 msgr2=0x7fc16c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.863+0000 7fc183fff640 1 --2- 172.21.15.69:0/3003911875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc16c003990 0x7fc16c003dd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0075a0 tx=0x7fc16c0057f0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.863+0000 7fc1837fe640 1 -- 172.21.15.69:0/3003911875 reap_dead start 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.863+0000 7fc183fff640 1 -- 172.21.15.69:0/3003911875 shutdown_connections 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.863+0000 7fc183fff640 1 -- 172.21.15.69:0/3003911875 >> 172.21.15.69:0/3003911875 conn(0x7fc17c0065e0 msgr2=0x7fc17c0069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.863+0000 7fc183fff640 1 -- 172.21.15.69:0/3003911875 shutdown_connections 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.863+0000 7fc183fff640 1 -- 172.21.15.69:0/3003911875 wait complete. 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.864+0000 7fc183fff640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.864+0000 7fc183fff640 10 monclient: monmap: 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.875 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.875 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.875 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.875 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.875 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.864+0000 7fc183fff640 1 librados: starting msgr at 2023-12-15T17:02:02.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.864+0000 7fc183fff640 1 librados: starting objecter 2023-12-15T17:02:02.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.864+0000 7f54348f4640 1 -- 172.21.15.69:0/1713454001 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542002d040 con 0x7f5430142bf0 2023-12-15T17:02:02.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.864+0000 7f54348f4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f54348f4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:02.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f54348f4640 10 monclient: dump: 2023-12-15T17:02:02.876 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.876 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.876 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.876 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.876 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f54348f4640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f54348f4640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f54348f4640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f54348f4640 1 -- 172.21.15.69:0/1713454001 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5420031810 con 0x7f5430142bf0 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f54348f4640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f54348f4640 1 -- 172.21.15.69:0/1713454001 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5420031ba0 con 0x7f5430142bf0 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f54348f4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f54348f4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f54348f4640 10 monclient: dump: 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.878 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f5437b5a640 5 monclient: authenticate success, global_id 4320 2023-12-15T17:02:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f5437b5a640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f5437b5a640 4 set_mon_vals no callback set 2023-12-15T17:02:02.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7fc183fff640 1 Processor -- start 2023-12-15T17:02:02.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7fc183fff640 1 -- start start 2023-12-15T17:02:02.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7fc183fff640 1 librados: setting wanted keys 2023-12-15T17:02:02.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7fc183fff640 1 librados: calling monclient init 2023-12-15T17:02:02.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7fc183fff640 10 monclient: init 2023-12-15T17:02:02.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7febfe4b3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febf81454a0 0x7febf8145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f5437b5a640 10 monclient: shutdown 2023-12-15T17:02:02.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f5437b5a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:02.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f5437b5a640 1 -- 172.21.15.69:0/1713454001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5430142bf0 msgr2=0x7f543013ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.865+0000 7f5437b5a640 1 --2- 172.21.15.69:0/1713454001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5430142bf0 0x7f543013ecf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f54200097a0 tx=0x7f542002b8b0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:02:02.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f5437b5a640 1 -- 172.21.15.69:0/1713454001 shutdown_connections 2023-12-15T17:02:02.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f5437b5a640 1 --2- 172.21.15.69:0/1713454001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5430142bf0 0x7f543013ecf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f5437b5a640 1 --2- 172.21.15.69:0/1713454001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54300a8e10 0x7f543013f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f5437b5a640 1 -- 172.21.15.69:0/1713454001 >> 172.21.15.69:0/1713454001 conn(0x7f5430061a30 msgr2=0x7f54300b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:02.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f52bd3e0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52b80b0f20 0x7f52b8124330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c003990 0x7fc17c0c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc16c003990 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc17c033250 0x7fc17c160480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 10 monclient(hunting): picked mon.noname-a con 0x7fc17c033250 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c1609c0 0x7fc17c031250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 10 monclient(hunting): picked mon.noname-b con 0x7fc17c1609c0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c0cbba0 con 0x7fc17c033250 2023-12-15T17:02:02.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c0be4e0 con 0x7fc17c1609c0 2023-12-15T17:02:02.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c160e40 con 0x7fc16c003990 2023-12-15T17:02:02.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7fc183fff640 10 monclient(hunting): authenticate will time out at 1427.182548s 2023-12-15T17:02:02.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f5437b5a640 1 -- 172.21.15.69:0/1713454001 shutdown_connections 2023-12-15T17:02:02.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f5437b5a640 1 -- 172.21.15.69:0/1713454001 wait complete. 2023-12-15T17:02:02.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f52bd3e0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52b80b0f20 0x7f52b8124330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58666/0 (socket says 172.21.15.69:58666) 2023-12-15T17:02:02.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f52bd3e0640 1 -- 172.21.15.69:0/1933679049 learned_addr learned my addr 172.21.15.69:0/1933679049 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f5437b5a640 10 monclient: build_initial_monmap 2023-12-15T17:02:02.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f5437b5a640 10 monclient: monmap: 2023-12-15T17:02:02.883 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:02.883 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:02.883 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:02.883 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:02.884 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:02.884 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.884 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:02.884 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:02.884 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:02.884 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f52bd3e0640 10 monclient(hunting): get_auth_request con 0x7f52b80b0f20 auth_method 0 2023-12-15T17:02:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f52bd3e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f52bd3e0640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.866+0000 7f52bd3e0640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.867+0000 7f5437b5a640 1 librados: starting msgr at 2023-12-15T17:02:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.867+0000 7f5437b5a640 1 librados: starting objecter 2023-12-15T17:02:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.867+0000 7f52bd3e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.867+0000 7f5437b5a640 1 Processor -- start 2023-12-15T17:02:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.867+0000 7f52bd3e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.867+0000 7f52bd3e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.867+0000 7f5437b5a640 1 -- start start 2023-12-15T17:02:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.867+0000 7f5437b5a640 1 librados: setting wanted keys 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.867+0000 7f5437b5a640 1 librados: calling monclient init 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.867+0000 7f5437b5a640 10 monclient: init 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54300a8a30 0x7f5430143df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 10 monclient(hunting): picked mon.noname-c con 0x7f54300a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54300a8e10 0x7f54300b58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 10 monclient(hunting): picked mon.noname-a con 0x7f54300a8e10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5430142bf0 0x7f54301d7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 10 monclient(hunting): picked mon.noname-b con 0x7f5430142bf0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54301d71e0 con 0x7f54300a8e10 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54300b77b0 con 0x7f5430142bf0 2023-12-15T17:02:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54301d7bb0 con 0x7f54300a8a30 2023-12-15T17:02:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.868+0000 7f5437b5a640 10 monclient(hunting): authenticate will time out at 1427.185345s 2023-12-15T17:02:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.871+0000 7febfe4b3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febf81454a0 0x7febf8145930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58664/0 (socket says 172.21.15.69:58664) 2023-12-15T17:02:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.871+0000 7febfe4b3640 1 -- 172.21.15.69:0/3436064748 learned_addr learned my addr 172.21.15.69:0/3436064748 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.871+0000 7febfe4b3640 10 monclient(hunting): get_auth_request con 0x7febf81454a0 auth_method 0 2023-12-15T17:02:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.871+0000 7febfe4b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.871+0000 7febfe4b3640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.871+0000 7febfe4b3640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.874+0000 7f52bd3e0640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2023-12-15T17:02:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.874+0000 7f52bd3e0640 1 -- 172.21.15.69:0/1933679049 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52b8143950 msgr2=0x7f52b813fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.874+0000 7f52bd3e0640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52b8143950 0x7f52b813fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.874+0000 7f52bd3e0640 1 -- 172.21.15.69:0/1933679049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52b80b29e0 msgr2=0x7f52b813f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.874+0000 7f52bd3e0640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52b80b29e0 0x7f52b813f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.874+0000 7f52bd3e0640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.874+0000 7f52bd3e0640 1 monclient: found mon.noname-a 2023-12-15T17:02:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.874+0000 7f52bd3e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.874+0000 7f52bd3e0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.874+0000 7f52bd3e0640 1 -- 172.21.15.69:0/1933679049 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b0009b30 con 0x7f52b80b0f20 2023-12-15T17:02:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.875+0000 7febfe4b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.875+0000 7febfe4b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.875+0000 7febfe4b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.878+0000 7f52bd3e0640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52b80b0f20 0x7f52b8124330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f52b400c480 tx=0x7f52b400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.878+0000 7f52adffb640 10 client.?.objecter ms_handle_connect 0x7f52b80b0f20 2023-12-15T17:02:02.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.878+0000 7f52adffb640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:02.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.879+0000 7febfe4b3640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2023-12-15T17:02:02.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.879+0000 7febfe4b3640 1 -- 172.21.15.69:0/3436064748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febf80a8a30 msgr2=0x7febf81424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.879+0000 7febfe4b3640 1 --2- 172.21.15.69:0/3436064748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febf80a8a30 0x7febf81424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.879+0000 7febfe4b3640 1 -- 172.21.15.69:0/3436064748 >> v2:172.21.15.192:3300/0 conn(0x7febf80a8e10 msgr2=0x7febf8142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.879+0000 7febfe4b3640 1 --2- 172.21.15.69:0/3436064748 >> v2:172.21.15.192:3300/0 conn(0x7febf80a8e10 0x7febf8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.879+0000 7febfe4b3640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.879+0000 7febfe4b3640 1 monclient: found mon.noname-a 2023-12-15T17:02:02.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.879+0000 7febfe4b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.879+0000 7febfe4b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.879+0000 7febfe4b3640 1 -- 172.21.15.69:0/3436064748 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf80a97f0 con 0x7febf81454a0 2023-12-15T17:02:02.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7fc1827fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc17c033250 0x7fc17c160480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:02.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2309213136 0 0) 0x7f690c003520 con 0x7f691c039160 2023-12-15T17:02:02.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 >> v2:172.21.15.192:3300/0 conn(0x7f691c0c8a30 msgr2=0x7f691c0d3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 1 --2- 172.21.15.69:0/467680622 >> v2:172.21.15.192:3300/0 conn(0x7f691c0c8a30 0x7f691c0d3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 >> v1:172.21.15.192:6789/0 conn(0x7f691c038d80 legacy=0x7f691c0c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:02.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 1 monclient: found mon.noname-a-legacy 2023-12-15T17:02:02.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.69:6789/0 2023-12-15T17:02:02.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c0c5940 con 0x7f691c039160 2023-12-15T17:02:02.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7fc1827fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc17c033250 0x7fc17c160480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58680/0 (socket says 172.21.15.69:58680) 2023-12-15T17:02:02.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7fc1827fc640 1 -- 172.21.15.69:0/4054546660 learned_addr learned my addr 172.21.15.69:0/4054546660 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:02.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7fc1827fc640 10 monclient(hunting): get_auth_request con 0x7fc17c033250 auth_method 0 2023-12-15T17:02:02.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7fc1827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:02.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7fc1827fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:02.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7fc1827fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:02.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2578675164 0 0) 0x7f6918001670 con 0x7f691c039160 2023-12-15T17:02:02.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692359e640 5 monclient: authenticate success, global_id 4323 2023-12-15T17:02:02.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-12-15T17:02:02.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 10 monclient: dump: 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] but i'm connected to v1:172.21.15.69:6789/0, reconnecting 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 >> v1:172.21.15.69:6789/0 conn(0x7f691c039160 legacy=0x7f691c0c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.884+0000 7f692159a640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 1 --2- 172.21.15.69:0/467680622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f690c0039c0 0x7f690c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 10 monclient(hunting): picked mon.b con 0x7f690c0039c0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 1 --2- 172.21.15.69:0/467680622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f690c0047c0 0x7f690c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 10 monclient(hunting): picked mon.c con 0x7f690c0047c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 1 --2- 172.21.15.69:0/467680622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f690c005330 0x7f690c00b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 10 monclient(hunting): picked mon.a con 0x7f690c005330 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c037610 con 0x7f690c005330 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c00c280 con 0x7f690c0039c0 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690c00c440 con 0x7f690c0047c0 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692159a640 10 monclient(hunting): _finish_auth 0 2023-12-15T17:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.885+0000 7f692359e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 1 -- 172.21.15.69:0/1933679049 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b40052d0 con 0x7f52b80b0f20 2023-12-15T17:02:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 10 monclient: dump: 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 10 monclient: _finish_auth 0 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 10 monclient: _check_auth_tickets 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 1 -- 172.21.15.69:0/1933679049 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b4005450 con 0x7f52b80b0f20 2023-12-15T17:02:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7febfe4b3640 1 --2- 172.21.15.69:0/3436064748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf81454a0 0x7febf8145930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7febf0009e50 tx=0x7febf002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954f5b0ec48b6409 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 1 -- 172.21.15.69:0/1933679049 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b4012870 con 0x7f52b80b0f20 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52adffb640 10 monclient: dump: 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:02.904 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52af7fe640 4 set_mon_vals no callback set 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52bf644640 5 monclient: authenticate success, global_id 4326 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52bf644640 10 monclient: _renew_subs 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52bf644640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52bf644640 1 -- 172.21.15.69:0/1933679049 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b81248d0 con 0x7f52b80b0f20 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52bf644640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52bf644640 10 monclient: _renew_subs 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52bf644640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52bf644640 1 -- 172.21.15.69:0/1933679049 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b8123980 con 0x7f52b80b0f20 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.886+0000 7f52bf644640 1 librados: init done 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.887+0000 7f52bf644640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.887+0000 7f52bf644640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.887+0000 7f52bf644640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.887+0000 7f52bf644640 1 -- 172.21.15.69:0/1933679049 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b8067ac0 con 0x7f52b80b0f20 2023-12-15T17:02:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.888+0000 7fc1827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.888+0000 7fc1827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.888+0000 7fc1827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.892+0000 7fc1827fc640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2023-12-15T17:02:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.892+0000 7fc1827fc640 1 -- 172.21.15.69:0/4054546660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c003990 msgr2=0x7fc17c0c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.892+0000 7fc1827fc640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c003990 0x7fc17c0c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.892+0000 7fc1827fc640 1 -- 172.21.15.69:0/4054546660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c1609c0 msgr2=0x7fc17c031250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.892+0000 7fc1827fc640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c1609c0 0x7fc17c031250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.892+0000 7fc1827fc640 10 monclient: _finish_hunting 0 2023-12-15T17:02:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.892+0000 7fc1827fc640 1 monclient: found mon.noname-a 2023-12-15T17:02:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.892+0000 7fc1827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.892+0000 7fc1827fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:02.892+0000 7fc1827fc640 1 -- 172.21.15.69:0/4054546660 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c0cb300 con 0x7fc17c033250 2023-12-15T17:02:03.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.033+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2802e050 con 0x7fba20003970 2023-12-15T17:02:03.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.033+0000 7fba35ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:03.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.033+0000 7fba35ffb640 1 -- 172.21.15.69:0/896063400 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba280329c0 con 0x7fba20003970 2023-12-15T17:02:03.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.034+0000 7fba35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.034+0000 7fba35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:03.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.034+0000 7fba35ffb640 10 monclient: dump: 2023-12-15T17:02:03.036 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.036 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.036 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.036 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.036 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.037 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.037 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.037 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.037 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.037 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.034+0000 7fba3d890640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:03.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.034+0000 7fba3d890640 4 set_mon_vals no callback set 2023-12-15T17:02:03.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.034+0000 7fba3d890640 10 monclient: shutdown 2023-12-15T17:02:03.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.034+0000 7fba3d890640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:03.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.034+0000 7fba3d890640 1 -- 172.21.15.69:0/896063400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba20003970 msgr2=0x7fba20003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.034+0000 7fba3d890640 1 --2- 172.21.15.69:0/896063400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba20003970 0x7fba20003d90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba28009860 tx=0x7fba200057f0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.034+0000 7fba377fe640 1 -- 172.21.15.69:0/896063400 reap_dead start 2023-12-15T17:02:03.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.035+0000 7fba3d890640 1 -- 172.21.15.69:0/896063400 shutdown_connections 2023-12-15T17:02:03.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.035+0000 7fba3d890640 1 -- 172.21.15.69:0/896063400 >> 172.21.15.69:0/896063400 conn(0x7fba3003a8f0 msgr2=0x7fba30036b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:03.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.035+0000 7fba3d890640 1 -- 172.21.15.69:0/896063400 shutdown_connections 2023-12-15T17:02:03.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.035+0000 7fba3d890640 1 -- 172.21.15.69:0/896063400 wait complete. 2023-12-15T17:02:03.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.036+0000 7fba3d890640 10 monclient: build_initial_monmap 2023-12-15T17:02:03.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.036+0000 7fba3d890640 10 monclient: monmap: 2023-12-15T17:02:03.040 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:03.040 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:03.040 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:03.040 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:03.040 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:03.040 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.040 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:03.040 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:03.040 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:03.041 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.036+0000 7fba3d890640 1 librados: starting msgr at 2023-12-15T17:02:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.036+0000 7fba3d890640 1 librados: starting objecter 2023-12-15T17:02:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.036+0000 7fba3d890640 1 Processor -- start 2023-12-15T17:02:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.036+0000 7fba3d890640 1 -- start start 2023-12-15T17:02:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.036+0000 7fba3d890640 1 librados: setting wanted keys 2023-12-15T17:02:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.036+0000 7fba3d890640 1 librados: calling monclient init 2023-12-15T17:02:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.036+0000 7fba3d890640 10 monclient: init 2023-12-15T17:02:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:02:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba20003970 0x7fba300c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 10 monclient(hunting): picked mon.noname-a con 0x7fba20003970 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba300b8f40 0x7fba3002dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 10 monclient(hunting): picked mon.noname-b con 0x7fba300b8f40 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba3002e210 0x7fba300c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 10 monclient(hunting): picked mon.noname-c con 0x7fba3002e210 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:03.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:03.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba300c71e0 con 0x7fba20003970 2023-12-15T17:02:03.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:03.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.037+0000 7fba3d890640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3002e650 con 0x7fba300b8f40 2023-12-15T17:02:03.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.038+0000 7fba3d890640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:03.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.038+0000 7fba3d890640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3002f3f0 con 0x7fba3002e210 2023-12-15T17:02:03.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.038+0000 7fba3d890640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:03.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.038+0000 7fba3d890640 10 monclient(hunting): authenticate will time out at 1427.354377s 2023-12-15T17:02:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.044+0000 7f89d2ffd640 1 -- 172.21.15.69:0/670731553 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f89cc054220 con 0x7f89d402ea70 2023-12-15T17:02:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.044+0000 7f89d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.044+0000 7f89d2ffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.050+0000 7fba367fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba300b8f40 0x7fba3002dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.051+0000 7f54360f7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5430142bf0 0x7f54301d7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.051+0000 7fba367fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba300b8f40 0x7fba3002dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37368/0 (socket says 172.21.15.69:37368) 2023-12-15T17:02:03.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.051+0000 7fba367fc640 1 -- 172.21.15.69:0/566353807 learned_addr learned my addr 172.21.15.69:0/566353807 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:03.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.051+0000 7fba367fc640 10 monclient(hunting): get_auth_request con 0x7fba300b8f40 auth_method 0 2023-12-15T17:02:03.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.051+0000 7fba367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:03.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.051+0000 7fba367fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:03.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.051+0000 7fba367fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:03.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.052+0000 7f692259c640 1 --2- 172.21.15.69:0/467680622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f690c0039c0 0x7f690c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f692259c640 10 monclient(hunting): get_auth_request con 0x7f690c0039c0 auth_method 0 2023-12-15T17:02:03.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f692259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:03.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f692259c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:03.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f692259c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-15T17:02:03.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7fba367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:03.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7fba367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:03.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7fba367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f54360f7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5430142bf0 0x7f54301d7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37344/0 (socket says 172.21.15.69:37344) 2023-12-15T17:02:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f54360f7640 1 -- 172.21.15.69:0/2486558344 learned_addr learned my addr 172.21.15.69:0/2486558344 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f54360f7640 10 monclient(hunting): get_auth_request con 0x7f5430142bf0 auth_method 0 2023-12-15T17:02:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f54360f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f54360f7640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f54360f7640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f692259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f692259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.053+0000 7f692259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-15T17:02:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.056+0000 7fba367fc640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2023-12-15T17:02:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.056+0000 7fba367fc640 1 -- 172.21.15.69:0/566353807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba3002e210 msgr2=0x7fba300c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.056+0000 7fba367fc640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba3002e210 0x7fba300c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.056+0000 7fba367fc640 1 -- 172.21.15.69:0/566353807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba20003970 msgr2=0x7fba300c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.056+0000 7fba367fc640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba20003970 0x7fba300c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.056+0000 7fba367fc640 10 monclient: _finish_hunting 0 2023-12-15T17:02:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.056+0000 7fba367fc640 1 monclient: found mon.noname-b 2023-12-15T17:02:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.056+0000 7fba367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.056+0000 7fba367fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.056+0000 7fba367fc640 1 -- 172.21.15.69:0/566353807 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3002f6d0 con 0x7fba300b8f40 2023-12-15T17:02:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.057+0000 7f54360f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.057+0000 7f54360f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.057+0000 7f54360f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 10 monclient(hunting): handle_auth_done global_id 4323 payload 293 2023-12-15T17:02:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 1 -- 172.21.15.69:0/467680622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f690c0047c0 msgr2=0x7f690c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 1 --2- 172.21.15.69:0/467680622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f690c0047c0 0x7f690c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 1 -- 172.21.15.69:0/467680622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f690c005330 msgr2=0x7f690c00b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 1 --2- 172.21.15.69:0/467680622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f690c005330 0x7f690c00b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 10 monclient: _finish_hunting 0 2023-12-15T17:02:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 1 monclient: found mon.b 2023-12-15T17:02:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 1 -- 172.21.15.69:0/467680622 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690c003520 con 0x7f690c0039c0 2023-12-15T17:02:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 10 monclient: _finish_auth 0 2023-12-15T17:02:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 10 monclient: _check_auth_tickets 2023-12-15T17:02:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f692259c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.060+0000 7f6922d9d640 1 -- 172.21.15.69:0/467680622 reap_dead start 2023-12-15T17:02:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.064+0000 7f54360f7640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2023-12-15T17:02:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.064+0000 7f54360f7640 1 -- 172.21.15.69:0/2486558344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54300a8a30 msgr2=0x7f5430143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.064+0000 7f54360f7640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54300a8a30 0x7f5430143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.064+0000 7f54360f7640 1 -- 172.21.15.69:0/2486558344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54300a8e10 msgr2=0x7f54300b58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.064+0000 7f54360f7640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54300a8e10 0x7f54300b58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.064+0000 7f54360f7640 10 monclient: _finish_hunting 0 2023-12-15T17:02:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.064+0000 7f54360f7640 1 monclient: found mon.noname-b 2023-12-15T17:02:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.064+0000 7f54360f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.064+0000 7f54360f7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.064+0000 7f54360f7640 1 -- 172.21.15.69:0/2486558344 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5420009400 con 0x7f5430142bf0 2023-12-15T17:02:03.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.068+0000 7fba367fc640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba300b8f40 0x7fba3002dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba38050140 tx=0x7fba38067880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.068+0000 7fba1ffff640 10 client.?.objecter ms_handle_connect 0x7fba300b8f40 2023-12-15T17:02:03.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.068+0000 7fba1ffff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:03.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.070+0000 7f54360f7640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5430142bf0 0x7f54301d7670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f542800b970 tx=0x7f542800be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.070+0000 7f54267fc640 10 client.?.objecter ms_handle_connect 0x7f5430142bf0 2023-12-15T17:02:03.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.070+0000 7f54267fc640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:03.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.072+0000 7f692259c640 1 --2- 172.21.15.69:0/467680622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f690c0039c0 0x7f690c003dc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6914001650 tx=0x7f690c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b972d13fdc4707ea server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 1 -- 172.21.15.69:0/566353807 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3806cd60 con 0x7fba300b8f40 2023-12-15T17:02:03.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:03.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 10 monclient: dump: 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 10 monclient: _finish_auth 0 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 10 monclient: _check_auth_tickets 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:03.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 1 -- 172.21.15.69:0/566353807 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba38072050 con 0x7fba300b8f40 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 1 -- 172.21.15.69:0/566353807 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba380723c0 con 0x7fba300b8f40 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba3d890640 5 monclient: authenticate success, global_id 4345 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba3d890640 10 monclient: _renew_subs 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba3d890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba3d890640 1 -- 172.21.15.69:0/566353807 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3002f9b0 con 0x7fba300b8f40 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba1ffff640 10 monclient: dump: 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.085 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba357fa640 4 set_mon_vals no callback set 2023-12-15T17:02:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba3d890640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba3d890640 10 monclient: _renew_subs 2023-12-15T17:02:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba3d890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba3d890640 1 -- 172.21.15.69:0/566353807 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba300b93a0 con 0x7fba300b8f40 2023-12-15T17:02:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7fba3d890640 1 librados: init done 2023-12-15T17:02:03.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914005270 con 0x7f690c0039c0 2023-12-15T17:02:03.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692159a640 10 monclient: dump: 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691402e070 con 0x7f690c0039c0 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:03.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692159a640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692359e640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692159a640 1 -- 172.21.15.69:0/467680622 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914003610 con 0x7f690c0039c0 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692359e640 4 set_mon_vals no callback set 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.080+0000 7f692159a640 10 monclient: dump: 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.088 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7f692359e640 10 monclient: shutdown 2023-12-15T17:02:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7f692359e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7f692359e640 1 -- 172.21.15.69:0/467680622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f690c0039c0 msgr2=0x7f690c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7f692359e640 1 --2- 172.21.15.69:0/467680622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f690c0039c0 0x7f690c003dc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6914001650 tx=0x7f690c0057f0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7f692359e640 1 -- 172.21.15.69:0/467680622 shutdown_connections 2023-12-15T17:02:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7f692359e640 1 --2- 172.21.15.69:0/467680622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f690c0039c0 0x7f690c003dc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7f692359e640 1 -- 172.21.15.69:0/467680622 >> 172.21.15.69:0/467680622 conn(0x7f691c0065e0 msgr2=0x7f691c0069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7f692359e640 1 -- 172.21.15.69:0/467680622 shutdown_connections 2023-12-15T17:02:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7f692359e640 1 -- 172.21.15.69:0/467680622 wait complete. 2023-12-15T17:02:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7fba3d890640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7fba3d890640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7fba3d890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.081+0000 7fba3d890640 1 -- 172.21.15.69:0/566353807 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3002e7c0 con 0x7fba300b8f40 2023-12-15T17:02:03.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 1 -- 172.21.15.69:0/2486558344 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542800cb40 con 0x7f5430142bf0 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 10 monclient: dump: 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 10 monclient: _finish_auth 0 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 10 monclient: _check_auth_tickets 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 1 -- 172.21.15.69:0/2486558344 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542800ccc0 con 0x7f5430142bf0 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 1 -- 172.21.15.69:0/2486558344 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428014680 con 0x7f5430142bf0 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f5437b5a640 5 monclient: authenticate success, global_id 4348 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f54267fc640 10 monclient: dump: 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f5437b5a640 10 monclient: _renew_subs 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f5437b5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f5427fff640 4 set_mon_vals no callback set 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f5437b5a640 1 -- 172.21.15.69:0/2486558344 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54301d7dd0 con 0x7f5430142bf0 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f5437b5a640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f5437b5a640 10 monclient: _renew_subs 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f5437b5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f5437b5a640 1 -- 172.21.15.69:0/2486558344 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5430144be0 con 0x7f5430142bf0 2023-12-15T17:02:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f5437b5a640 1 librados: init done 2023-12-15T17:02:03.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f692359e640 10 monclient: build_initial_monmap 2023-12-15T17:02:03.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f692359e640 10 monclient: monmap: 2023-12-15T17:02:03.094 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:03.094 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:03.094 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:03.094 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:03.094 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:03.094 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.094 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:03.095 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:03.095 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:03.095 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f692359e640 1 librados: starting msgr at 2023-12-15T17:02:03.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.082+0000 7f692359e640 1 librados: starting objecter 2023-12-15T17:02:03.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 1 Processor -- start 2023-12-15T17:02:03.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 1 -- start start 2023-12-15T17:02:03.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 1 librados: setting wanted keys 2023-12-15T17:02:03.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 1 librados: calling monclient init 2023-12-15T17:02:03.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 10 monclient: init 2023-12-15T17:02:03.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f5437b5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f5437b5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f5437b5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f5437b5a640 1 -- 172.21.15.69:0/2486558344 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5430067ac0 con 0x7f5430142bf0 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f690c0039c0 0x7f691c0caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 10 monclient(hunting): picked mon.noname-b con 0x7f690c0039c0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f691c02fcc0 0x7f691c0c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 10 monclient(hunting): picked mon.noname-c con 0x7f691c02fcc0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f691c15d390 0x7f691c15d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 10 monclient(hunting): picked mon.noname-a con 0x7f691c15d390 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c15cf70 con 0x7f691c15d390 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0c9af0 con 0x7f690c0039c0 2023-12-15T17:02:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:03.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0c9f00 con 0x7f691c02fcc0 2023-12-15T17:02:03.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:03.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.083+0000 7f692359e640 10 monclient(hunting): authenticate will time out at 1427.400230s 2023-12-15T17:02:03.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.094+0000 7f6eed7fa640 1 -- 172.21.15.69:0/1021920638 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6ee8058890 con 0x7f6ef0071b70 2023-12-15T17:02:03.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.094+0000 7f6eed7fa640 10 client.4295.objecter ms_dispatch 0x7f6ef0000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:03.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.094+0000 7f6eed7fa640 3 client.4295.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:03.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.094+0000 7f6eed7fa640 3 client.4295.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:03.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.094+0000 7f6eed7fa640 20 client.4295.objecter dump_active .. 0 homeless 2023-12-15T17:02:03.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.094+0000 7f6eed7fa640 1 -- 172.21.15.69:0/1021920638 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6ee8054bd0 con 0x7f6ef0071b70 2023-12-15T17:02:03.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.094+0000 7f6eed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.094+0000 7f6eed7fa640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:03.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.108+0000 7f6921d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f691c02fcc0 0x7f691c0c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.108+0000 7f54267fc640 1 -- 172.21.15.69:0/2486558344 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f542801d3b0 con 0x7f5430142bf0 2023-12-15T17:02:03.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.108+0000 7f54267fc640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f541406cb50 0x7f541406f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.109+0000 7f54358f6640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f541406cb50 0x7f541406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.109+0000 7f54358f6640 10 monclient: get_auth_request con 0x7f541406cb50 auth_method 0 2023-12-15T17:02:03.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.109+0000 7f54358f6640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f541406cb50 0x7f541406f010 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f542c001610 tx=0x7f542c007930 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.109+0000 7f54267fc640 10 client.4348.objecter ms_handle_connect 0x7f541406cb50 2023-12-15T17:02:03.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.109+0000 7f6921d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f691c02fcc0 0x7f691c0c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53014/0 (socket says 172.21.15.69:53014) 2023-12-15T17:02:03.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.109+0000 7f6921d9b640 1 -- 172.21.15.69:0/1201458920 learned_addr learned my addr 172.21.15.69:0/1201458920 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:03.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.109+0000 7f6921d9b640 10 monclient(hunting): get_auth_request con 0x7f691c02fcc0 auth_method 0 2023-12-15T17:02:03.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.109+0000 7f6921d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:03.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.109+0000 7f6921d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:03.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.109+0000 7f6921d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:03.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.111+0000 7f6921d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:03.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.111+0000 7f6921d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:03.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.111+0000 7f6921d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:03.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.115+0000 7f6921d9b640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2023-12-15T17:02:03.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.115+0000 7f6921d9b640 1 -- 172.21.15.69:0/1201458920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f690c0039c0 msgr2=0x7f691c0caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.115+0000 7f6921d9b640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f690c0039c0 0x7f691c0caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.115+0000 7f6921d9b640 1 -- 172.21.15.69:0/1201458920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f691c15d390 msgr2=0x7f691c15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.115+0000 7f6921d9b640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f691c15d390 0x7f691c15d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.115+0000 7f6921d9b640 10 monclient: _finish_hunting 0 2023-12-15T17:02:03.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.115+0000 7f6921d9b640 1 monclient: found mon.noname-c 2023-12-15T17:02:03.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.115+0000 7f6921d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:03.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.115+0000 7f6921d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:02:03.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.115+0000 7f6921d9b640 1 -- 172.21.15.69:0/1201458920 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c02eef0 con 0x7f691c02fcc0 2023-12-15T17:02:03.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.118+0000 7f6921d9b640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f691c02fcc0 0x7f691c0c95b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f691800c7d0 tx=0x7f691800cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.118+0000 7f6912ffd640 10 client.?.objecter ms_handle_connect 0x7f691c02fcc0 2023-12-15T17:02:03.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.118+0000 7f6912ffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:03.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 1 -- 172.21.15.69:0/1201458920 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69180074e0 con 0x7f691c02fcc0 2023-12-15T17:02:03.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:03.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 10 monclient: dump: 2023-12-15T17:02:03.127 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.127 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 1 -- 172.21.15.69:0/1201458920 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6918007640 con 0x7f691c02fcc0 2023-12-15T17:02:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 1 -- 172.21.15.69:0/1201458920 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918004a20 con 0x7f691c02fcc0 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6912ffd640 10 monclient: dump: 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.129 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f692359e640 5 monclient: authenticate success, global_id 4301 2023-12-15T17:02:03.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f6920d99640 4 set_mon_vals no callback set 2023-12-15T17:02:03.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f692359e640 10 monclient: _renew_subs 2023-12-15T17:02:03.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f692359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:03.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.124+0000 7f692359e640 1 -- 172.21.15.69:0/1201458920 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c0303b0 con 0x7f691c02fcc0 2023-12-15T17:02:03.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.125+0000 7f692359e640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:03.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.125+0000 7f692359e640 10 monclient: _renew_subs 2023-12-15T17:02:03.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.125+0000 7f692359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:03.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.125+0000 7f692359e640 1 -- 172.21.15.69:0/1201458920 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c0cb8e0 con 0x7f691c02fcc0 2023-12-15T17:02:03.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.125+0000 7f692359e640 1 librados: init done 2023-12-15T17:02:03.131 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.126+0000 7f692359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.131 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.126+0000 7f692359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.131 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.126+0000 7f692359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:03.131 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.126+0000 7f692359e640 1 -- 172.21.15.69:0/1201458920 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c15ccd0 con 0x7f691c02fcc0 2023-12-15T17:02:03.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.151+0000 7f6912ffd640 1 -- 172.21.15.69:0/1201458920 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f691800a640 con 0x7f691c02fcc0 2023-12-15T17:02:03.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.151+0000 7f6912ffd640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f690406cbd0 0x7f690406f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.151+0000 7f692259c640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f690406cbd0 0x7f690406f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.151+0000 7f692259c640 10 monclient: get_auth_request con 0x7f690406cbd0 auth_method 0 2023-12-15T17:02:03.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.152+0000 7f6912ffd640 10 client.4301.objecter ms_handle_connect 0x7f690406cbd0 2023-12-15T17:02:03.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.152+0000 7f692259c640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f690406cbd0 0x7f690406f090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6914033000 tx=0x7f6914005000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.160+0000 7f52adffb640 1 -- 172.21.15.69:0/1933679049 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f52b40129d0 con 0x7f52b80b0f20 2023-12-15T17:02:03.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.160+0000 7f52adffb640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f529c06cbd0 0x7f529c06f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.160+0000 7f52bcbdf640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f529c06cbd0 0x7f529c06f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.160+0000 7f52bcbdf640 10 monclient: get_auth_request con 0x7f529c06cbd0 auth_method 0 2023-12-15T17:02:03.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.161+0000 7f52adffb640 10 client.4326.objecter ms_handle_connect 0x7f529c06cbd0 2023-12-15T17:02:03.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.161+0000 7f52bcbdf640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f529c06cbd0 0x7f529c06f090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52b80b3ed0 tx=0x7f52a8007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.162+0000 7fc1827fc640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc17c033250 0x7fc17c160480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc17400c6f0 tx=0x7fc17400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.162+0000 7fc16bfff640 10 client.?.objecter ms_handle_connect 0x7fc17c033250 2023-12-15T17:02:03.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.162+0000 7fc16bfff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:03.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.177+0000 7f52adffb640 1 -- 172.21.15.69:0/1933679049 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f52b4058f50 con 0x7f52b80b0f20 2023-12-15T17:02:03.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.177+0000 7f52adffb640 10 client.4326.objecter ms_dispatch 0x7f52b8000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:03.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.177+0000 7f52adffb640 3 client.4326.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:03.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.177+0000 7f52adffb640 3 client.4326.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:03.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.177+0000 7f52adffb640 20 client.4326.objecter dump_active .. 0 homeless 2023-12-15T17:02:03.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.177+0000 7f52adffb640 1 -- 172.21.15.69:0/1933679049 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f52b4054b90 con 0x7f52b80b0f20 2023-12-15T17:02:03.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.177+0000 7f52adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.177+0000 7f52adffb640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:03.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 1 -- 172.21.15.69:0/3436064748 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf002ca80 con 0x7febf81454a0 2023-12-15T17:02:03.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:03.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 10 monclient: dump: 2023-12-15T17:02:03.189 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.189 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 10 monclient: _finish_auth 0 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 10 monclient: _check_auth_tickets 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 1 -- 172.21.15.69:0/3436064748 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf002cbe0 con 0x7febf81454a0 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:03.190 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 1 -- 172.21.15.69:0/3436064748 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf00343f0 con 0x7febf81454a0 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfccb0640 10 monclient: dump: 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfff16640 5 monclient: authenticate success, global_id 4329 2023-12-15T17:02:03.191 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfff16640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:03.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.187+0000 7febfff16640 4 set_mon_vals no callback set 2023-12-15T17:02:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.188+0000 7febfff16640 10 monclient: shutdown 2023-12-15T17:02:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.188+0000 7febfff16640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.188+0000 7febfff16640 1 -- 172.21.15.69:0/3436064748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf81454a0 msgr2=0x7febf8145930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.188+0000 7febfff16640 1 --2- 172.21.15.69:0/3436064748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf81454a0 0x7febf8145930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7febf0009e50 tx=0x7febf002bee0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.188+0000 7febfff16640 1 -- 172.21.15.69:0/3436064748 shutdown_connections 2023-12-15T17:02:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.188+0000 7febfff16640 1 --2- 172.21.15.69:0/3436064748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf81454a0 0x7febf8145930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.188+0000 7febfff16640 1 --2- 172.21.15.69:0/3436064748 >> v2:172.21.15.192:3300/0 conn(0x7febf80a8e10 0x7febf8142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.188+0000 7febfff16640 1 --2- 172.21.15.69:0/3436064748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febf80a8a30 0x7febf81424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.188+0000 7febfff16640 1 -- 172.21.15.69:0/3436064748 >> 172.21.15.69:0/3436064748 conn(0x7febf8061a30 msgr2=0x7febf80b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.188+0000 7febfff16640 1 -- 172.21.15.69:0/3436064748 shutdown_connections 2023-12-15T17:02:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.188+0000 7febfff16640 1 -- 172.21.15.69:0/3436064748 wait complete. 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.189+0000 7febfff16640 10 monclient: build_initial_monmap 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.189+0000 7febfff16640 10 monclient: monmap: 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.189+0000 7febfff16640 1 librados: starting msgr at 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.189+0000 7febfff16640 1 librados: starting objecter 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.189+0000 7febfff16640 1 Processor -- start 2023-12-15T17:02:03.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.189+0000 7febfff16640 1 -- start start 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.189+0000 7febfff16640 1 librados: setting wanted keys 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.189+0000 7febfff16640 1 librados: calling monclient init 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.189+0000 7febfff16640 10 monclient: init 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febf80a8a30 0x7febf81441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 10 monclient(hunting): picked mon.noname-c con 0x7febf80a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febf80a8e10 0x7febf80ab2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 10 monclient(hunting): picked mon.noname-b con 0x7febf80a8e10 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf81454a0 0x7febf81399a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 10 monclient(hunting): picked mon.noname-a con 0x7febf81454a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf80ac220 con 0x7febf81454a0 2023-12-15T17:02:03.196 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:03.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf81449d0 con 0x7febf80a8e10 2023-12-15T17:02:03.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:03.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf80abd00 con 0x7febf80a8a30 2023-12-15T17:02:03.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:03.197 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.190+0000 7febfff16640 10 monclient(hunting): authenticate will time out at 1427.507094s 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 1 -- 172.21.15.69:0/4054546660 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1740072d0 con 0x7fc17c033250 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 10 monclient: dump: 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.199 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 10 monclient: _finish_auth 0 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 10 monclient: _check_auth_tickets 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc183fff640 5 monclient: authenticate success, global_id 4332 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc183fff640 10 monclient: _renew_subs 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc183fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc183fff640 1 -- 172.21.15.69:0/4054546660 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c032680 con 0x7fc17c033250 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 1 -- 172.21.15.69:0/4054546660 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc174007450 con 0x7fc17c033250 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 1 -- 172.21.15.69:0/4054546660 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1740047e0 con 0x7fc17c033250 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc16bfff640 10 monclient: dump: 2023-12-15T17:02:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc183fff640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc183fff640 10 monclient: _renew_subs 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc183fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc183fff640 1 -- 172.21.15.69:0/4054546660 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c0cbfc0 con 0x7fc17c033250 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc183fff640 1 librados: init done 2023-12-15T17:02:03.201 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.193+0000 7fc1817fa640 4 set_mon_vals no callback set 2023-12-15T17:02:03.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.195+0000 7fc183fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.195+0000 7fc183fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.195+0000 7fc183fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:03.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.195+0000 7fc183fff640 1 -- 172.21.15.69:0/4054546660 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17c0cb900 con 0x7fc17c033250 2023-12-15T17:02:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.273+0000 7f89e332c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-15T17:02:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.273+0000 7f89e332c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-15T17:02:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.273+0000 7f89e332c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.273+0000 7f89e332c640 1 -- 172.21.15.69:0/670731553 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f89c40055c0 con 0x7f89d402ea70 2023-12-15T17:02:03.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.302+0000 7f6ef6c7c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-15T17:02:03.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.302+0000 7f6ef6c7c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-15T17:02:03.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.302+0000 7f6ef6c7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:03.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.302+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/1021920638 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f6ed00055c0 con 0x7f6ef0071b70 2023-12-15T17:02:03.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.358+0000 7f52bf644640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-15T17:02:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.358+0000 7f52bf644640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-15T17:02:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.358+0000 7f52bf644640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.358+0000 7f52bf644640 1 -- 172.21.15.69:0/1933679049 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f52b8143d80 con 0x7f52b80b0f20 2023-12-15T17:02:03.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.472+0000 7febfe4b3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf81454a0 0x7febf81399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.486+0000 7f6912ffd640 1 -- 172.21.15.69:0/1201458920 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6918058780 con 0x7f691c02fcc0 2023-12-15T17:02:03.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.486+0000 7f6912ffd640 10 client.4301.objecter ms_dispatch 0x7f6924000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:03.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.486+0000 7f6912ffd640 3 client.4301.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:03.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.486+0000 7f6912ffd640 3 client.4301.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:03.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.486+0000 7f6912ffd640 20 client.4301.objecter dump_active .. 0 homeless 2023-12-15T17:02:03.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.486+0000 7f6912ffd640 1 -- 172.21.15.69:0/1201458920 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f69180543c0 con 0x7f691c02fcc0 2023-12-15T17:02:03.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.486+0000 7f6912ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.486+0000 7f6912ffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:03.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.490+0000 7febfdcb2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febf80a8a30 0x7febf81441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.491+0000 7febfdcb2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febf80a8a30 0x7febf81441a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53030/0 (socket says 172.21.15.69:53030) 2023-12-15T17:02:03.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.491+0000 7febfdcb2640 1 -- 172.21.15.69:0/3664247912 learned_addr learned my addr 172.21.15.69:0/3664247912 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:03.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.491+0000 7febfdcb2640 10 monclient(hunting): get_auth_request con 0x7febf80a8a30 auth_method 0 2023-12-15T17:02:03.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.491+0000 7febfdcb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:03.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.491+0000 7febfdcb2640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:03.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.491+0000 7febfdcb2640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:03.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.493+0000 7fc16bfff640 1 -- 172.21.15.69:0/4054546660 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc174004960 con 0x7fc17c033250 2023-12-15T17:02:03.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.494+0000 7fc16bfff640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc17006cba0 0x7fc17006f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.494+0000 7fc182ffd640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc17006cba0 0x7fc17006f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.494+0000 7fc182ffd640 10 monclient: get_auth_request con 0x7fc17006cba0 auth_method 0 2023-12-15T17:02:03.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.494+0000 7fc16bfff640 10 client.4332.objecter ms_handle_connect 0x7fc17006cba0 2023-12-15T17:02:03.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.494+0000 7fc182ffd640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc17006cba0 0x7fc17006f060 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc178034000 tx=0x7fc17802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.498+0000 7febfe4b3640 10 monclient(hunting): get_auth_request con 0x7febf81454a0 auth_method 0 2023-12-15T17:02:03.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.498+0000 7febfe4b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:03.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.498+0000 7febfe4b3640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:03.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.498+0000 7febfe4b3640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:03.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.498+0000 7febfdcb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:03.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.498+0000 7febfdcb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:03.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.498+0000 7febfdcb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:03.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.499+0000 7f52adffb640 1 -- 172.21.15.69:0/1933679049 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f52b8143d80 con 0x7f52b80b0f20 2023-12-15T17:02:03.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.499+0000 7f52adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-15T17:02:03.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.499+0000 7f52adffb640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:03.501 INFO:teuthology.orchestra.run.smithi069.stdout:64424509443 2023-12-15T17:02:03.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.500+0000 7f6eed7fa640 1 -- 172.21.15.69:0/1021920638 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6ee8017020 con 0x7f6ef0071b70 2023-12-15T17:02:03.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.500+0000 7f6eed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-15T17:02:03.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.500+0000 7f6eed7fa640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:03.502 INFO:teuthology.orchestra.run.smithi069.stdout:60129542148 2023-12-15T17:02:03.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 10 librados: watch_flush enter 2023-12-15T17:02:03.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 10 librados: watch_flush exit 2023-12-15T17:02:03.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 20 client.4326.objecter shutdown clearing up homeless session... 2023-12-15T17:02:03.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 10 client.4326.objecter successfully canceled tick 2023-12-15T17:02:03.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 -- 172.21.15.69:0/1933679049 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f529c06cbd0 msgr2=0x7f529c06f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f529c06cbd0 0x7f529c06f090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52b80b3ed0 tx=0x7f52a8007990 comp rx=0 tx=0).stop 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 10 monclient: shutdown 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 -- 172.21.15.69:0/1933679049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52b80b0f20 msgr2=0x7f52b8124330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52b80b0f20 0x7f52b8124330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f52b400c480 tx=0x7f52b400c940 comp rx=0 tx=0).stop 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 -- 172.21.15.69:0/1933679049 shutdown_connections 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f529c06cbd0 0x7f529c06f090 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52b8143950 0x7f52b813fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52b80b29e0 0x7f52b813f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 --2- 172.21.15.69:0/1933679049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52b80b0f20 0x7f52b8124330 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 -- 172.21.15.69:0/1933679049 >> 172.21.15.69:0/1933679049 conn(0x7f52b8061a30 msgr2=0x7f52b8141880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 -- 172.21.15.69:0/1933679049 shutdown_connections 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 -- 172.21.15.69:0/1933679049 wait complete. 2023-12-15T17:02:03.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.501+0000 7f52bf644640 1 librados: shutdown 2023-12-15T17:02:03.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.502+0000 7febfe4b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:03.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.502+0000 7febfe4b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:03.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.502+0000 7febfe4b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:03.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 10 librados: watch_flush enter 2023-12-15T17:02:03.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 10 librados: watch_flush exit 2023-12-15T17:02:03.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 20 client.4295.objecter shutdown clearing up homeless session... 2023-12-15T17:02:03.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 10 client.4295.objecter successfully canceled tick 2023-12-15T17:02:03.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/1021920638 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6edc06cba0 msgr2=0x7f6edc06f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6edc06cba0 0x7f6edc06f060 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400bb00 tx=0x7f6ee4008040 comp rx=0 tx=0).stop 2023-12-15T17:02:03.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 10 monclient: shutdown 2023-12-15T17:02:03.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:03.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/1021920638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef0071b70 msgr2=0x7f6ef01f4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef0071b70 0x7f6ef01f4950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6ee800cbc0 tx=0x7f6ee800a430 comp rx=0 tx=0).stop 2023-12-15T17:02:03.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/1021920638 shutdown_connections 2023-12-15T17:02:03.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6edc06cba0 0x7f6edc06f060 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef00af360 0x7f6ef01f4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ef0071b70 0x7f6ef01f4950 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 --2- 172.21.15.69:0/1021920638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ef006fbc0 0x7f6ef0088b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/1021920638 >> 172.21.15.69:0/1021920638 conn(0x7f6ef00b01e0 msgr2=0x7f6ef008a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:03.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/1021920638 shutdown_connections 2023-12-15T17:02:03.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 -- 172.21.15.69:0/1021920638 wait complete. 2023-12-15T17:02:03.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.503+0000 7f6ef6c7c640 1 librados: shutdown 2023-12-15T17:02:03.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.504+0000 7febfdcb2640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2023-12-15T17:02:03.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.504+0000 7febfdcb2640 1 -- 172.21.15.69:0/3664247912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febf80a8e10 msgr2=0x7febf80ab2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.504+0000 7febfdcb2640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febf80a8e10 0x7febf80ab2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.504+0000 7febfdcb2640 1 -- 172.21.15.69:0/3664247912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf81454a0 msgr2=0x7febf81399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.504+0000 7febfdcb2640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf81454a0 0x7febf81399a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.504+0000 7febfdcb2640 10 monclient: _finish_hunting 0 2023-12-15T17:02:03.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.504+0000 7febfdcb2640 1 monclient: found mon.noname-c 2023-12-15T17:02:03.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.504+0000 7febfdcb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:03.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.504+0000 7febfdcb2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:02:03.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.504+0000 7febfdcb2640 1 -- 172.21.15.69:0/3664247912 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf0009b30 con 0x7febf80a8a30 2023-12-15T17:02:03.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.510+0000 7febfdcb2640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febf80a8a30 0x7febf81441a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7febf400b4a0 tx=0x7febf400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.510+0000 7febee7fc640 10 client.?.objecter ms_handle_connect 0x7febf80a8a30 2023-12-15T17:02:03.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.510+0000 7febee7fc640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:03.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.513+0000 7fc16bfff640 1 -- 172.21.15.69:0/4054546660 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc174058670 con 0x7fc17c033250 2023-12-15T17:02:03.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.513+0000 7fc16bfff640 10 client.4332.objecter ms_dispatch 0x7fc184000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:03.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.513+0000 7fc16bfff640 3 client.4332.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:03.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.513+0000 7fc16bfff640 3 client.4332.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:03.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.513+0000 7fc16bfff640 20 client.4332.objecter dump_active .. 0 homeless 2023-12-15T17:02:03.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.513+0000 7fc16bfff640 1 -- 172.21.15.69:0/4054546660 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc1740542b0 con 0x7fc17c033250 2023-12-15T17:02:03.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.514+0000 7fc16bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.514+0000 7fc16bfff640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:03.518 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509443 for osd.2 2023-12-15T17:02:03.519 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.7 2023-12-15T17:02:03.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 1 -- 172.21.15.69:0/3664247912 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf4010610 con 0x7febf80a8a30 2023-12-15T17:02:03.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:03.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 10 monclient: dump: 2023-12-15T17:02:03.519 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.519 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.519 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.519 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 10 monclient: _finish_auth 0 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 10 monclient: _check_auth_tickets 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 1 -- 172.21.15.69:0/3664247912 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf4010770 con 0x7febf80a8a30 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febfff16640 5 monclient: authenticate success, global_id 4304 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febfff16640 10 monclient: _renew_subs 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febfff16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febfff16640 1 -- 172.21.15.69:0/3664247912 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf80ab810 con 0x7febf80a8a30 2023-12-15T17:02:03.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:03.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 1 -- 172.21.15.69:0/3664247912 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf4010b20 con 0x7febf80a8a30 2023-12-15T17:02:03.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:03.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:03.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febee7fc640 10 monclient: dump: 2023-12-15T17:02:03.521 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:03.521 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febeffff640 4 set_mon_vals no callback set 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febfff16640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febfff16640 10 monclient: _renew_subs 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febfff16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febfff16640 1 -- 172.21.15.69:0/3664247912 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf813a0c0 con 0x7febf80a8a30 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.517+0000 7febfff16640 1 librados: init done 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.518+0000 7febfff16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.518+0000 7febfff16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.518+0000 7febfff16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:03.523 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.518+0000 7febfff16640 1 -- 172.21.15.69:0/3664247912 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf8067ac0 con 0x7febf80a8a30 2023-12-15T17:02:03.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.550+0000 7febee7fc640 1 -- 172.21.15.69:0/3664247912 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7febf4010c80 con 0x7febf80a8a30 2023-12-15T17:02:03.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.550+0000 7febee7fc640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7febdc06cb50 0x7febdc06f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.550+0000 7febfd4b1640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7febdc06cb50 0x7febdc06f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.550+0000 7febfd4b1640 10 monclient: get_auth_request con 0x7febdc06cb50 auth_method 0 2023-12-15T17:02:03.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.551+0000 7febee7fc640 10 client.4304.objecter ms_handle_connect 0x7febdc06cb50 2023-12-15T17:02:03.552 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.551+0000 7febfd4b1640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7febdc06cb50 0x7febdc06f010 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7febf81d4810 tx=0x7febe8006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.556+0000 7febee7fc640 1 -- 172.21.15.69:0/3664247912 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7febf4058490 con 0x7febf80a8a30 2023-12-15T17:02:03.558 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.556+0000 7febee7fc640 10 client.4304.objecter ms_dispatch 0x7febf8000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:03.558 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.556+0000 7febee7fc640 3 client.4304.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:03.558 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.556+0000 7febee7fc640 3 client.4304.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:03.558 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.556+0000 7febee7fc640 20 client.4304.objecter dump_active .. 0 homeless 2023-12-15T17:02:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.565+0000 7fba1ffff640 1 -- 172.21.15.69:0/566353807 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fba38072520 con 0x7fba300b8f40 2023-12-15T17:02:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.565+0000 7fba1ffff640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fba2406cba0 0x7fba2406f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.565+0000 7fba36ffd640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fba2406cba0 0x7fba2406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.565+0000 7fba36ffd640 10 monclient: get_auth_request con 0x7fba2406cba0 auth_method 0 2023-12-15T17:02:03.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.566+0000 7fba1ffff640 10 client.4345.objecter ms_handle_connect 0x7fba2406cba0 2023-12-15T17:02:03.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.566+0000 7fba36ffd640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fba2406cba0 0x7fba2406f060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba28034000 tx=0x7fba2802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:03.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.567+0000 7f54267fc640 1 -- 172.21.15.69:0/2486558344 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f542800ce30 con 0x7f5430142bf0 2023-12-15T17:02:03.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.567+0000 7f54267fc640 10 client.4348.objecter ms_dispatch 0x7f5430000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:03.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.567+0000 7f54267fc640 3 client.4348.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:03.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.567+0000 7f54267fc640 3 client.4348.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:03.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.567+0000 7f54267fc640 20 client.4348.objecter dump_active .. 0 homeless 2023-12-15T17:02:03.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.576+0000 7f54267fc640 1 -- 172.21.15.69:0/2486558344 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f5428058630 con 0x7f5430142bf0 2023-12-15T17:02:03.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.576+0000 7f54267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.576+0000 7f54267fc640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:03.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.582+0000 7fba1ffff640 1 -- 172.21.15.69:0/566353807 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fba380902f0 con 0x7fba300b8f40 2023-12-15T17:02:03.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.582+0000 7fba1ffff640 10 client.4345.objecter ms_dispatch 0x7fba38000b90 osd_map(22..22 src has 1..22) v4 2023-12-15T17:02:03.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.582+0000 7fba1ffff640 3 client.4345.objecter handle_osd_map got epochs [22,22] > 0 2023-12-15T17:02:03.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.582+0000 7fba1ffff640 3 client.4345.objecter handle_osd_map decoding full epoch 22 2023-12-15T17:02:03.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.582+0000 7fba1ffff640 20 client.4345.objecter dump_active .. 0 homeless 2023-12-15T17:02:03.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.592+0000 7fba1ffff640 1 -- 172.21.15.69:0/566353807 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fba380ced90 con 0x7fba300b8f40 2023-12-15T17:02:03.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.592+0000 7fba1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.592+0000 7fba1ffff640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.619+0000 7f89d2ffd640 1 -- 172.21.15.69:0/670731553 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f89cc017020 con 0x7f89d402ea70 2023-12-15T17:02:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.619+0000 7f89d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-15T17:02:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.619+0000 7f89d2ffd640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:03.622 INFO:teuthology.orchestra.run.smithi069.stdout:64424509445 2023-12-15T17:02:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.622+0000 7f89e332c640 10 librados: watch_flush enter 2023-12-15T17:02:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.622+0000 7f89e332c640 10 librados: watch_flush exit 2023-12-15T17:02:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.622+0000 7f89e332c640 20 client.4336.objecter shutdown clearing up homeless session... 2023-12-15T17:02:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.622+0000 7f89e332c640 10 client.4336.objecter successfully canceled tick 2023-12-15T17:02:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.622+0000 7f89e332c640 1 -- 172.21.15.69:0/670731553 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f89c006cbd0 msgr2=0x7f89c006f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.622+0000 7f89e332c640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f89c006cbd0 0x7f89c006f090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f89dc068a20 tx=0x7f89dc06e040 comp rx=0 tx=0).stop 2023-12-15T17:02:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.622+0000 7f89e332c640 10 monclient: shutdown 2023-12-15T17:02:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.622+0000 7f89e332c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.622+0000 7f89e332c640 1 -- 172.21.15.69:0/670731553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d402ea70 msgr2=0x7f89d4031c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.622+0000 7f89e332c640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d402ea70 0x7f89d4031c10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00c410 tx=0x7f89cc00c8d0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.623+0000 7f89e332c640 1 -- 172.21.15.69:0/670731553 shutdown_connections 2023-12-15T17:02:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.623+0000 7f89e332c640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f89c006cbd0 0x7f89c006f090 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.623+0000 7f89e332c640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89d40c48c0 0x7f89d4169f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.623+0000 7f89e332c640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d402ea70 0x7f89d4031c10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.623+0000 7f89e332c640 1 --2- 172.21.15.69:0/670731553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d402de00 0x7f89d4030f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.623+0000 7f89e332c640 1 -- 172.21.15.69:0/670731553 >> 172.21.15.69:0/670731553 conn(0x7f89d4005eb0 msgr2=0x7f89d40c3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.623+0000 7f89e332c640 1 -- 172.21.15.69:0/670731553 shutdown_connections 2023-12-15T17:02:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.623+0000 7f89e332c640 1 -- 172.21.15.69:0/670731553 wait complete. 2023-12-15T17:02:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.623+0000 7f89e332c640 1 librados: shutdown 2023-12-15T17:02:03.640 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.0 2023-12-15T17:02:03.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.737+0000 7f692359e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-15T17:02:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.737+0000 7f692359e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-15T17:02:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.737+0000 7f692359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.737+0000 7f692359e640 1 -- 172.21.15.69:0/1201458920 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f691c15d0b0 con 0x7f691c02fcc0 2023-12-15T17:02:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.788+0000 7f5437b5a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-15T17:02:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.788+0000 7f5437b5a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-15T17:02:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.788+0000 7f5437b5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.788+0000 7f5437b5a640 1 -- 172.21.15.69:0/2486558344 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f54301d6fc0 con 0x7f5430142bf0 2023-12-15T17:02:03.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.792+0000 7f54267fc640 1 -- 172.21.15.69:0/2486558344 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5428019070 con 0x7f5430142bf0 2023-12-15T17:02:03.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.792+0000 7f54267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-15T17:02:03.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.792+0000 7f54267fc640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:03.794 INFO:teuthology.orchestra.run.smithi069.stdout:60129542149 2023-12-15T17:02:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 10 librados: watch_flush enter 2023-12-15T17:02:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 10 librados: watch_flush exit 2023-12-15T17:02:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 20 client.4348.objecter shutdown clearing up homeless session... 2023-12-15T17:02:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 10 client.4348.objecter successfully canceled tick 2023-12-15T17:02:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 -- 172.21.15.69:0/2486558344 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f541406cb50 msgr2=0x7f541406f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f541406cb50 0x7f541406f010 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f542c001610 tx=0x7f542c007930 comp rx=0 tx=0).stop 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 10 monclient: shutdown 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 -- 172.21.15.69:0/2486558344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5430142bf0 msgr2=0x7f54301d7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5430142bf0 0x7f54301d7670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f542800b970 tx=0x7f542800be30 comp rx=0 tx=0).stop 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 -- 172.21.15.69:0/2486558344 shutdown_connections 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f541406cb50 0x7f541406f010 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5430142bf0 0x7f54301d7670 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54300a8e10 0x7f54300b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 --2- 172.21.15.69:0/2486558344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54300a8a30 0x7f5430143df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 -- 172.21.15.69:0/2486558344 >> 172.21.15.69:0/2486558344 conn(0x7f5430061a30 msgr2=0x7f54301396c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 -- 172.21.15.69:0/2486558344 shutdown_connections 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 -- 172.21.15.69:0/2486558344 wait complete. 2023-12-15T17:02:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.795+0000 7f5437b5a640 1 librados: shutdown 2023-12-15T17:02:03.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.6 2023-12-15T17:02:03.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.823+0000 7fc183fff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-15T17:02:03.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.823+0000 7fc183fff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-15T17:02:03.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.823+0000 7fc183fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:03.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.823+0000 7fc183fff640 1 -- 172.21.15.69:0/4054546660 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc17c0cbce0 con 0x7fc17c033250 2023-12-15T17:02:03.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.829+0000 7fc16bfff640 1 -- 172.21.15.69:0/4054546660 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc17c0cbce0 con 0x7fc17c033250 2023-12-15T17:02:03.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.829+0000 7fc16bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-15T17:02:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.829+0000 7fc16bfff640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:03.831 INFO:teuthology.orchestra.run.smithi069.stdout:60129542150 2023-12-15T17:02:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.829+0000 7febee7fc640 1 -- 172.21.15.69:0/3664247912 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7febf40547d0 con 0x7febf80a8a30 2023-12-15T17:02:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.829+0000 7febee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.829+0000 7febee7fc640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 10 librados: watch_flush enter 2023-12-15T17:02:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 10 librados: watch_flush exit 2023-12-15T17:02:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 20 client.4332.objecter shutdown clearing up homeless session... 2023-12-15T17:02:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 10 client.4332.objecter successfully canceled tick 2023-12-15T17:02:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 -- 172.21.15.69:0/4054546660 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc17006cba0 msgr2=0x7fc17006f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc17006cba0 0x7fc17006f060 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc178034000 tx=0x7fc17802f000 comp rx=0 tx=0).stop 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 10 monclient: shutdown 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 -- 172.21.15.69:0/4054546660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc17c033250 msgr2=0x7fc17c160480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc17c033250 0x7fc17c160480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc17400c6f0 tx=0x7fc17400cbb0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 -- 172.21.15.69:0/4054546660 shutdown_connections 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc17006cba0 0x7fc17006f060 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c1609c0 0x7fc17c031250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc17c033250 0x7fc17c160480 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 --2- 172.21.15.69:0/4054546660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c003990 0x7fc17c0c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 -- 172.21.15.69:0/4054546660 >> 172.21.15.69:0/4054546660 conn(0x7fc17c0065e0 msgr2=0x7fc17c0c4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 -- 172.21.15.69:0/4054546660 shutdown_connections 2023-12-15T17:02:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 -- 172.21.15.69:0/4054546660 wait complete. 2023-12-15T17:02:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.831+0000 7fc183fff640 1 librados: shutdown 2023-12-15T17:02:03.844 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.5 2023-12-15T17:02:03.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.902+0000 7f6912ffd640 1 -- 172.21.15.69:0/1201458920 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f691c15d0b0 con 0x7f691c02fcc0 2023-12-15T17:02:03.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.902+0000 7f6912ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-15T17:02:03.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.902+0000 7f6912ffd640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:03.905 INFO:teuthology.orchestra.run.smithi069.stdout:64424509444 2023-12-15T17:02:03.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 10 librados: watch_flush enter 2023-12-15T17:02:03.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 10 librados: watch_flush exit 2023-12-15T17:02:03.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 20 client.4301.objecter shutdown clearing up homeless session... 2023-12-15T17:02:03.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 10 client.4301.objecter successfully canceled tick 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 -- 172.21.15.69:0/1201458920 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f690406cbd0 msgr2=0x7f690406f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f690406cbd0 0x7f690406f090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6914033000 tx=0x7f6914005000 comp rx=0 tx=0).stop 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 10 monclient: shutdown 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 -- 172.21.15.69:0/1201458920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f691c02fcc0 msgr2=0x7f691c0c95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f691c02fcc0 0x7f691c0c95b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f691800c7d0 tx=0x7f691800cc90 comp rx=0 tx=0).stop 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 -- 172.21.15.69:0/1201458920 shutdown_connections 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f690406cbd0 0x7f690406f090 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f691c15d390 0x7f691c15d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f691c02fcc0 0x7f691c0c95b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 --2- 172.21.15.69:0/1201458920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f690c0039c0 0x7f691c0caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 -- 172.21.15.69:0/1201458920 >> 172.21.15.69:0/1201458920 conn(0x7f691c0065e0 msgr2=0x7f691c0c4910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 -- 172.21.15.69:0/1201458920 shutdown_connections 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 -- 172.21.15.69:0/1201458920 wait complete. 2023-12-15T17:02:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.905+0000 7f692359e640 1 librados: shutdown 2023-12-15T17:02:03.919 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.3 2023-12-15T17:02:03.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.922+0000 7fba3d890640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-15T17:02:03.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.922+0000 7fba3d890640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-15T17:02:03.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.922+0000 7fba3d890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:03.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.922+0000 7fba3d890640 1 -- 172.21.15.69:0/566353807 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fba300056e0 con 0x7fba300b8f40 2023-12-15T17:02:03.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.976+0000 7fba1ffff640 1 -- 172.21.15.69:0/566353807 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fba380d23b0 con 0x7fba300b8f40 2023-12-15T17:02:03.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.976+0000 7fba1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-15T17:02:03.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.976+0000 7fba1ffff640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:03.978 INFO:teuthology.orchestra.run.smithi069.stdout:60129542150 2023-12-15T17:02:03.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.977+0000 7fba3d890640 10 librados: watch_flush enter 2023-12-15T17:02:03.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.977+0000 7fba3d890640 10 librados: watch_flush exit 2023-12-15T17:02:03.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.977+0000 7fba3d890640 20 client.4345.objecter shutdown clearing up homeless session... 2023-12-15T17:02:03.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.977+0000 7fba3d890640 10 client.4345.objecter successfully canceled tick 2023-12-15T17:02:03.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.977+0000 7fba3d890640 1 -- 172.21.15.69:0/566353807 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fba2406cba0 msgr2=0x7fba2406f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fba2406cba0 0x7fba2406f060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba28034000 tx=0x7fba2802f000 comp rx=0 tx=0).stop 2023-12-15T17:02:03.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 10 monclient: shutdown 2023-12-15T17:02:03.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:03.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 -- 172.21.15.69:0/566353807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba300b8f40 msgr2=0x7fba3002dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:03.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba300b8f40 0x7fba3002dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba38050140 tx=0x7fba38067880 comp rx=0 tx=0).stop 2023-12-15T17:02:03.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 -- 172.21.15.69:0/566353807 shutdown_connections 2023-12-15T17:02:03.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fba2406cba0 0x7fba2406f060 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba3002e210 0x7fba300c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba300b8f40 0x7fba3002dcd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 --2- 172.21.15.69:0/566353807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba20003970 0x7fba300c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:03.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 -- 172.21.15.69:0/566353807 >> 172.21.15.69:0/566353807 conn(0x7fba3003a8f0 msgr2=0x7fba300396d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:03.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 -- 172.21.15.69:0/566353807 shutdown_connections 2023-12-15T17:02:03.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 -- 172.21.15.69:0/566353807 wait complete. 2023-12-15T17:02:03.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:03.978+0000 7fba3d890640 1 librados: shutdown 2023-12-15T17:02:03.990 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.4 2023-12-15T17:02:04.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.039+0000 7febfff16640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-15T17:02:04.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.039+0000 7febfff16640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-15T17:02:04.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.039+0000 7febfff16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:04.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.039+0000 7febfff16640 1 -- 172.21.15.69:0/3664247912 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7febf8075b60 con 0x7febf80a8a30 2023-12-15T17:02:04.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.043+0000 7febee7fc640 1 -- 172.21.15.69:0/3664247912 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7febf401e020 con 0x7febf80a8a30 2023-12-15T17:02:04.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.043+0000 7febee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-15T17:02:04.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.043+0000 7febee7fc640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:04.046 INFO:teuthology.orchestra.run.smithi069.stdout:64424509445 2023-12-15T17:02:04.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 10 librados: watch_flush enter 2023-12-15T17:02:04.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 10 librados: watch_flush exit 2023-12-15T17:02:04.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 20 client.4304.objecter shutdown clearing up homeless session... 2023-12-15T17:02:04.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 10 client.4304.objecter successfully canceled tick 2023-12-15T17:02:04.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 -- 172.21.15.69:0/3664247912 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7febdc06cb50 msgr2=0x7febdc06f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7febdc06cb50 0x7febdc06f010 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7febf81d4810 tx=0x7febe8006c90 comp rx=0 tx=0).stop 2023-12-15T17:02:04.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 10 monclient: shutdown 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 -- 172.21.15.69:0/3664247912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febf80a8a30 msgr2=0x7febf81441a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febf80a8a30 0x7febf81441a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7febf400b4a0 tx=0x7febf400b960 comp rx=0 tx=0).stop 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 -- 172.21.15.69:0/3664247912 shutdown_connections 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7febdc06cb50 0x7febdc06f010 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf81454a0 0x7febf81399a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febf80a8e10 0x7febf80ab2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 --2- 172.21.15.69:0/3664247912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febf80a8a30 0x7febf81441a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 -- 172.21.15.69:0/3664247912 >> 172.21.15.69:0/3664247912 conn(0x7febf8061a30 msgr2=0x7febf8140840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 -- 172.21.15.69:0/3664247912 shutdown_connections 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 -- 172.21.15.69:0/3664247912 wait complete. 2023-12-15T17:02:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.046+0000 7febfff16640 1 librados: shutdown 2023-12-15T17:02:04.062 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.1 2023-12-15T17:02:04.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-15T17:02:04.520 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-15T17:02:04.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.622+0000 7fcbc9a39640 10 monclient: get_monmap_and_config 2023-12-15T17:02:04.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.622+0000 7fcbc9a39640 10 monclient: build_initial_monmap 2023-12-15T17:02:04.624 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.622+0000 7fcbc9a39640 10 monclient: monmap: 2023-12-15T17:02:04.624 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:04.624 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.624 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:04.623661+0000 2023-12-15T17:02:04.624 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:04.623661+0000 2023-12-15T17:02:04.624 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:04.624 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.624 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:04.624 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.622+0000 7f72af96c640 10 monclient: get_monmap_and_config 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.622+0000 7f72af96c640 10 monclient: build_initial_monmap 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.622+0000 7f72af96c640 10 monclient: monmap: 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:04.623695+0000 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:04.623695+0000 2023-12-15T17:02:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:04.626 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.622+0000 7fcbc9a39640 1 Processor -- start 2023-12-15T17:02:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.622+0000 7f72af96c640 1 Processor -- start 2023-12-15T17:02:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.622+0000 7fcbc9a39640 1 -- start start 2023-12-15T17:02:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.622+0000 7fcbc9a39640 10 monclient: init 2023-12-15T17:02:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 1 -- start start 2023-12-15T17:02:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 10 monclient: init 2023-12-15T17:02:04.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:04.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-12-15T17:02:04.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 0x7fcbc41424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 10 monclient(hunting): picked mon.noname-b con 0x7fcbc40a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:04.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcbc40a8e10 0x7fcbc4142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 10 monclient(hunting): picked mon.noname-c con 0x7fcbc40a8e10 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:04.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcbc41454a0 0x7fcbc4145930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 10 monclient(hunting): picked mon.noname-a con 0x7fcbc41454a0 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:04.628 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc40aaa10 con 0x7fcbc41454a0 2023-12-15T17:02:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc40aac10 con 0x7fcbc40a8e10 2023-12-15T17:02:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40aaef0 con 0x7fcbc40a8a30 2023-12-15T17:02:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7fcbc9a39640 10 monclient(hunting): authenticate will time out at 1428.939786s 2023-12-15T17:02:04.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:04.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-12-15T17:02:04.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f72a80a8a30 0x7f72a81424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 10 monclient(hunting): picked mon.noname-a con 0x7f72a80a8a30 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:04.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72a80a8e10 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:04.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72a8142a20 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:04.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a8142e70 con 0x7f72a8142a20 2023-12-15T17:02:04.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a81462c0 con 0x7f72a80a8e10 2023-12-15T17:02:04.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.630 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a80aaa10 con 0x7f72a80a8a30 2023-12-15T17:02:04.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:04.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.623+0000 7f72af96c640 10 monclient(hunting): authenticate will time out at 1428.940034s 2023-12-15T17:02:04.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 0x7fcbc41424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc27fc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcbc40a8e10 0x7fcbc4142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 0x7fcbc41424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53042/0 (socket says 172.21.15.69:53042) 2023-12-15T17:02:04.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc2ffd640 1 -- 172.21.15.69:0/2978332092 learned_addr learned my addr 172.21.15.69:0/2978332092 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:04.631 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc2ffd640 10 monclient(hunting): get_auth_request con 0x7fcbc40a8a30 auth_method 0 2023-12-15T17:02:04.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7f72acf07640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f72a80a8e10 0x7f72a81454a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:60592/0 (socket says 172.21.15.69:60592) 2023-12-15T17:02:04.632 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7f72acf07640 1 -- 172.21.15.69:0/2289935626 learned_addr learned my addr 172.21.15.69:0/2289935626 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:04.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc27fc640 10 monclient(hunting): get_auth_request con 0x7fcbc40a8e10 auth_method 0 2023-12-15T17:02:04.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc27fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.624+0000 7fcbc27fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.625+0000 7fcbc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.625+0000 7fcbc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.625+0000 7fcbc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.625+0000 7fcbc27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.625+0000 7fcbc27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.625+0000 7fcbc27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.626+0000 7fcbc37fe640 1 --2- 172.21.15.69:0/2978332092 >> v2:172.21.15.69:3300/0 conn(0x7fcbc41454a0 0x7fcbc4145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc37fe640 10 monclient(hunting): get_auth_request con 0x7fcbc41454a0 auth_method 0 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc37fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc2ffd640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2023-12-15T17:02:04.634 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc2ffd640 1 -- 172.21.15.69:0/2978332092 >> v2:172.21.15.192:3300/0 conn(0x7fcbc40a8e10 msgr2=0x7fcbc4142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:04.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc2ffd640 1 --2- 172.21.15.69:0/2978332092 >> v2:172.21.15.192:3300/0 conn(0x7fcbc40a8e10 0x7fcbc4142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc2ffd640 1 -- 172.21.15.69:0/2978332092 >> v2:172.21.15.69:3300/0 conn(0x7fcbc41454a0 msgr2=0x7fcbc4145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:04.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc2ffd640 1 --2- 172.21.15.69:0/2978332092 >> v2:172.21.15.69:3300/0 conn(0x7fcbc41454a0 0x7fcbc4145930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc2ffd640 10 monclient: _finish_hunting 0 2023-12-15T17:02:04.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc2ffd640 1 monclient: found mon.noname-b 2023-12-15T17:02:04.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:04.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:04.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7fcbc2ffd640 1 -- 172.21.15.69:0/2978332092 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc40a97f0 con 0x7fcbc40a8a30 2023-12-15T17:02:04.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.627+0000 7f72ad708640 1 --2- 172.21.15.69:0/2289935626 >> v2:172.21.15.69:3300/0 conn(0x7f72a80a8a30 0x7f72a81424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.628+0000 7f72ad708640 10 monclient(hunting): get_auth_request con 0x7f72a80a8a30 auth_method 0 2023-12-15T17:02:04.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.628+0000 7f72ad708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.628+0000 7f72ad708640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.628+0000 7f72ad708640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.629+0000 7fcbc2ffd640 1 --2- 172.21.15.69:0/2978332092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 0x7fcbc41424e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009870 tx=0x7fcbb402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33117656329aee8a server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:04.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.631+0000 7f72ad708640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.631+0000 7f72ad708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.631+0000 7f72ad708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.631+0000 7f729ffff640 1 -- 172.21.15.69:0/2289935626 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815374285 0 0) 0x7f72a81462c0 con 0x7f72a80a8e10 2023-12-15T17:02:04.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.631+0000 7f729ffff640 10 monclient(hunting): _negotiate 2023-12-15T17:02:04.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.631+0000 7f729ffff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.631+0000 7f729ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.631+0000 7f729ffff640 10 monclient(hunting): my global_id is 4354 2023-12-15T17:02:04.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.631+0000 7f729ffff640 1 -- 172.21.15.69:0/2289935626 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7294003520 con 0x7f72a80a8e10 2023-12-15T17:02:04.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 1 -- 172.21.15.69:0/2978332092 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4031640 con 0x7fcbc40a8a30 2023-12-15T17:02:04.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:04.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 10 monclient: dump: 2023-12-15T17:02:04.638 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.638 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.638 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.638 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.638 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.638 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 10 monclient: _finish_auth 0 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 10 monclient: _check_auth_tickets 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 1 -- 172.21.15.69:0/2978332092 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb40317a0 con 0x7fcbc40a8a30 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc9a39640 5 monclient: authenticate success, global_id 4307 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc9a39640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc9a39640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 1 -- 172.21.15.69:0/2978332092 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4031b30 con 0x7fcbc40a8a30 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc9a39640 4 set_mon_vals no callback set 2023-12-15T17:02:04.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7fcbc1ffb640 10 monclient: dump: 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.640 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.641 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7f72ad708640 10 monclient(hunting): handle_auth_done global_id 4341 payload 274 2023-12-15T17:02:04.641 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7f72ad708640 1 -- 172.21.15.69:0/2289935626 >> v1:172.21.15.192:6789/0 conn(0x7f72a80a8e10 legacy=0x7f72a81454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.641 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7f72ad708640 1 -- 172.21.15.69:0/2289935626 >> v1:172.21.15.69:6789/0 conn(0x7f72a8142a20 legacy=0x7f72a8145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:04.641 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7f72ad708640 10 monclient: _finish_hunting 0 2023-12-15T17:02:04.641 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7f72ad708640 1 monclient: found mon.noname-a 2023-12-15T17:02:04.641 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7f72ad708640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:04.641 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7f72ad708640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:04.641 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.633+0000 7f72ad708640 1 -- 172.21.15.69:0/2289935626 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a80aac10 con 0x7f72a80a8a30 2023-12-15T17:02:04.642 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-15T17:02:04.644 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.634+0000 7fcbc9a39640 10 monclient: shutdown 2023-12-15T17:02:04.644 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.634+0000 7fcbc9a39640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:04.644 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.634+0000 7fcbc9a39640 1 -- 172.21.15.69:0/2978332092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 msgr2=0x7fcbc41424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.644 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.634+0000 7fcbc9a39640 1 --2- 172.21.15.69:0/2978332092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 0x7fcbc41424e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009870 tx=0x7fcbb402b8f0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.644 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.634+0000 7fcbc9a39640 1 -- 172.21.15.69:0/2978332092 shutdown_connections 2023-12-15T17:02:04.644 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.634+0000 7fcbc9a39640 1 --2- 172.21.15.69:0/2978332092 >> v2:172.21.15.69:3300/0 conn(0x7fcbc41454a0 0x7fcbc4145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.644 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.634+0000 7fcbc9a39640 1 --2- 172.21.15.69:0/2978332092 >> v2:172.21.15.192:3300/0 conn(0x7fcbc40a8e10 0x7fcbc4142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.644 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.634+0000 7fcbc9a39640 1 --2- 172.21.15.69:0/2978332092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 0x7fcbc41424e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.634+0000 7fcbc9a39640 1 -- 172.21.15.69:0/2978332092 >> 172.21.15.69:0/2978332092 conn(0x7fcbc4061a30 msgr2=0x7fcbc40b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.634+0000 7fcbc9a39640 1 -- 172.21.15.69:0/2978332092 shutdown_connections 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.634+0000 7fcbc9a39640 1 -- 172.21.15.69:0/2978332092 wait complete. 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.635+0000 7fcbc9a39640 10 monclient: build_initial_monmap 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.635+0000 7fcbc9a39640 10 monclient: monmap: 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:04.645 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.635+0000 7fcbc9a39640 1 librados: starting msgr at 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.635+0000 7fcbc9a39640 1 librados: starting objecter 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.635+0000 7fcbc9a39640 1 Processor -- start 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.635+0000 7fcbc9a39640 1 -- start start 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.635+0000 7fcbc9a39640 1 librados: setting wanted keys 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.635+0000 7fcbc9a39640 1 librados: calling monclient init 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.635+0000 7fcbc9a39640 10 monclient: init 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 0x7fcbc40aec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 10 monclient(hunting): picked mon.noname-c con 0x7fcbc40a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbc40a8e10 0x7fcbc40af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 10 monclient(hunting): picked mon.noname-b con 0x7fcbc40a8e10 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc41454a0 0x7fcbc41dac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 10 monclient(hunting): picked mon.noname-a con 0x7fcbc41454a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:04.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc41da7e0 con 0x7fcbc41454a0 2023-12-15T17:02:04.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40adc60 con 0x7fcbc40a8e10 2023-12-15T17:02:04.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40af1c0 con 0x7fcbc40a8a30 2023-12-15T17:02:04.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:04.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7fcbc9a39640 10 monclient(hunting): authenticate will time out at 1428.952797s 2023-12-15T17:02:04.647 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.636+0000 7f72ad708640 1 --2- 172.21.15.69:0/2289935626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a80a8a30 0x7f72a81424e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f72a40097a0 tx=0x7f72a402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f33a8f8548c04f server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:04.649 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.637+0000 7fcbc27fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbc40a8e10 0x7fcbc40af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.649 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.637+0000 7fcbc27fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbc40a8e10 0x7fcbc40af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37408/0 (socket says 172.21.15.69:37408) 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.637+0000 7fcbc27fc640 1 -- 172.21.15.69:0/861125557 learned_addr learned my addr 172.21.15.69:0/861125557 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.637+0000 7fcbc27fc640 10 monclient(hunting): get_auth_request con 0x7fcbc40a8e10 auth_method 0 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.637+0000 7fcbc27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.637+0000 7fcbc27fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.637+0000 7fcbc27fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.637+0000 7fcbc2ffd640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 0x7fcbc40aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.637+0000 7fcbc27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.637+0000 7fcbc27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.637+0000 7fcbc27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.638+0000 7fcbc2ffd640 10 monclient(hunting): get_auth_request con 0x7fcbc40a8a30 auth_method 0 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.638+0000 7fcbc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.638+0000 7fcbc2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.638+0000 7fcbc2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.638+0000 7fcbc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.638+0000 7fcbc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.638+0000 7fcbc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.639+0000 7fcbc37fe640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc41454a0 0x7fcbc41dac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.640+0000 7fcbc27fc640 10 monclient(hunting): handle_auth_done global_id 4357 payload 824 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.640+0000 7fcbc27fc640 1 -- 172.21.15.69:0/861125557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 msgr2=0x7fcbc40aec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.640+0000 7fcbc27fc640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 0x7fcbc40aec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.640+0000 7fcbc27fc640 1 -- 172.21.15.69:0/861125557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc41454a0 msgr2=0x7fcbc41dac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.640+0000 7fcbc27fc640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc41454a0 0x7fcbc41dac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.640+0000 7fcbc27fc640 10 monclient: _finish_hunting 0 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.640+0000 7fcbc27fc640 1 monclient: found mon.noname-b 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.640+0000 7fcbc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.640+0000 7fcbc27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.640+0000 7fcbc27fc640 1 -- 172.21.15.69:0/861125557 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb40094d0 con 0x7fcbc40a8e10 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7fcbc27fc640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbc40a8e10 0x7fcbc40af6e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800e890 tx=0x7fcbb800ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7fcbabfff640 10 client.?.objecter ms_handle_connect 0x7fcbc40a8e10 2023-12-15T17:02:04.652 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7fcbabfff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 1 -- 172.21.15.69:0/2289935626 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a4030610 con 0x7f72a80a8a30 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 10 monclient: dump: 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.655 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 10 monclient: _finish_auth 0 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 10 monclient: _check_auth_tickets 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 1 -- 172.21.15.69:0/2289935626 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a4030770 con 0x7f72a80a8a30 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f72af96c640 5 monclient: authenticate success, global_id 4341 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f72af96c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f72af96c640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 1 -- 172.21.15.69:0/2289935626 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a4030b00 con 0x7f72a80a8a30 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f72af96c640 4 set_mon_vals no callback set 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:04.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.643+0000 7f729ffff640 10 monclient: dump: 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.644+0000 7f72af96c640 10 monclient: shutdown 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.644+0000 7f72af96c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.644+0000 7f72af96c640 1 -- 172.21.15.69:0/2289935626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a80a8a30 msgr2=0x7f72a81424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.644+0000 7f72af96c640 1 --2- 172.21.15.69:0/2289935626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a80a8a30 0x7f72a81424e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f72a40097a0 tx=0x7f72a402b960 comp rx=0 tx=0).stop 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.644+0000 7f72af96c640 1 -- 172.21.15.69:0/2289935626 shutdown_connections 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.644+0000 7f72af96c640 1 --2- 172.21.15.69:0/2289935626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a80a8a30 0x7f72a81424e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.644+0000 7f72af96c640 1 -- 172.21.15.69:0/2289935626 >> 172.21.15.69:0/2289935626 conn(0x7f72a8061a30 msgr2=0x7f72a80b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.644+0000 7f72af96c640 1 -- 172.21.15.69:0/2289935626 shutdown_connections 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.644+0000 7f72af96c640 1 -- 172.21.15.69:0/2289935626 wait complete. 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.645+0000 7f72af96c640 10 monclient: build_initial_monmap 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.645+0000 7f72af96c640 10 monclient: monmap: 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:04.658 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.645+0000 7f72af96c640 1 librados: starting msgr at 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.645+0000 7f72af96c640 1 librados: starting objecter 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.645+0000 7f72af96c640 1 Processor -- start 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.645+0000 7f72af96c640 1 -- start start 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.645+0000 7f72af96c640 1 librados: setting wanted keys 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.645+0000 7f72af96c640 1 librados: calling monclient init 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.645+0000 7f72af96c640 10 monclient: init 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72a80a8a30 0x7f72a81da600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 10 monclient(hunting): picked mon.noname-b con 0x7f72a80a8a30 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72a80a8e10 0x7f72a8144840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 10 monclient(hunting): picked mon.noname-c con 0x7f72a80a8e10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:04.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a8142a20 0x7f72a8144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 10 monclient(hunting): picked mon.noname-a con 0x7f72a8142a20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a80ae840 con 0x7f72a8142a20 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a80ab530 con 0x7f72a80a8a30 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a81dac00 con 0x7f72a80a8e10 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.646+0000 7f72af96c640 10 monclient(hunting): authenticate will time out at 1428.962609s 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.647+0000 7f72adf09640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a8142a20 0x7f72a8144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.647+0000 7f72adf09640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a8142a20 0x7f72a8144d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58748/0 (socket says 172.21.15.69:58748) 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.647+0000 7f72adf09640 1 -- 172.21.15.69:0/2577791817 learned_addr learned my addr 172.21.15.69:0/2577791817 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.647+0000 7f72adf09640 10 monclient(hunting): get_auth_request con 0x7f72a8142a20 auth_method 0 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.647+0000 7f72adf09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.647+0000 7f72adf09640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.661 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.647+0000 7f72adf09640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.661 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.648+0000 7f72adf09640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.661 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.648+0000 7f72adf09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.661 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.648+0000 7f72adf09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.661 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.648+0000 7f72acf07640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72a80a8e10 0x7f72a8144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.662 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.648+0000 7f72ad708640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72a80a8a30 0x7f72a81da600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.662 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7f72ad708640 10 monclient(hunting): get_auth_request con 0x7f72a80a8a30 auth_method 0 2023-12-15T17:02:04.662 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7f72ad708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.662 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7f72ad708640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.662 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7f72ad708640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.662 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7f72acf07640 10 monclient(hunting): get_auth_request con 0x7f72a80a8e10 auth_method 0 2023-12-15T17:02:04.662 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7f72acf07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.662 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7f72acf07640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.662 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7f72acf07640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.663 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 1 -- 172.21.15.69:0/861125557 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb800c940 con 0x7fcbc40a8e10 2023-12-15T17:02:04.663 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.663 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 10 monclient: dump: 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 10 monclient: _finish_auth 0 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 10 monclient: _check_auth_tickets 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:04.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 1 -- 172.21.15.69:0/861125557 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb800caa0 con 0x7fcbc40a8e10 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 1 -- 172.21.15.69:0/861125557 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb800ce30 con 0x7fcbc40a8e10 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbc9a39640 5 monclient: authenticate success, global_id 4357 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbc9a39640 10 monclient: _renew_subs 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbc9a39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbc9a39640 1 -- 172.21.15.69:0/861125557 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc40afc20 con 0x7fcbc40a8e10 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbc17fa640 4 set_mon_vals no callback set 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.649+0000 7fcbabfff640 10 monclient: dump: 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.666 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.666 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.666 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.666 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.666 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.650+0000 7fcbc9a39640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:04.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.650+0000 7fcbc9a39640 10 monclient: _renew_subs 2023-12-15T17:02:04.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.650+0000 7fcbc9a39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:04.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.650+0000 7fcbc9a39640 1 -- 172.21.15.69:0/861125557 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc41db140 con 0x7fcbc40a8e10 2023-12-15T17:02:04.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.650+0000 7fcbc9a39640 1 librados: init done 2023-12-15T17:02:04.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.650+0000 7f72acf07640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.650+0000 7f72acf07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.650+0000 7f72acf07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.650+0000 7f72ad708640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.650+0000 7f72ad708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.650+0000 7f72ad708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7fcbc9a39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7fcbc9a39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7fcbc9a39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7fcbc9a39640 1 -- 172.21.15.69:0/861125557 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc4067ac0 con 0x7fcbc40a8e10 2023-12-15T17:02:04.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7f72adf09640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2023-12-15T17:02:04.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7f72adf09640 1 -- 172.21.15.69:0/2577791817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72a80a8e10 msgr2=0x7f72a8144840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7f72adf09640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72a80a8e10 0x7f72a8144840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7f72adf09640 1 -- 172.21.15.69:0/2577791817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72a80a8a30 msgr2=0x7f72a81da600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7f72adf09640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72a80a8a30 0x7f72a81da600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7f72adf09640 10 monclient: _finish_hunting 0 2023-12-15T17:02:04.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7f72adf09640 1 monclient: found mon.noname-a 2023-12-15T17:02:04.668 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7f72adf09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:04.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7f72adf09640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:04.669 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.651+0000 7f72adf09640 1 -- 172.21.15.69:0/2577791817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4009400 con 0x7f72a8142a20 2023-12-15T17:02:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.702+0000 7f72adf09640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a8142a20 0x7f72a8144d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f72a000b970 tx=0x7f72a000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:04.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.702+0000 7f729dffb640 10 client.?.objecter ms_handle_connect 0x7f72a8142a20 2023-12-15T17:02:04.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.702+0000 7f729dffb640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:04.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 1 -- 172.21.15.69:0/2577791817 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a000cb40 con 0x7f72a8142a20 2023-12-15T17:02:04.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:04.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 10 monclient: dump: 2023-12-15T17:02:04.715 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 10 monclient: _finish_auth 0 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 10 monclient: _check_auth_tickets 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 1 -- 172.21.15.69:0/2577791817 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a000cca0 con 0x7f72a8142a20 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f72af96c640 5 monclient: authenticate success, global_id 4344 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 1 -- 172.21.15.69:0/2577791817 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a0014680 con 0x7f72a8142a20 2023-12-15T17:02:04.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f72af96c640 10 monclient: _renew_subs 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f72af96c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f72af96c640 1 -- 172.21.15.69:0/2577791817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a81db120 con 0x7f72a8142a20 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729f7fe640 4 set_mon_vals no callback set 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f729dffb640 10 monclient: dump: 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.717 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.718 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f72af96c640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:04.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f72af96c640 10 monclient: _renew_subs 2023-12-15T17:02:04.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f72af96c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:04.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f72af96c640 1 -- 172.21.15.69:0/2577791817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a81389b0 con 0x7f72a8142a20 2023-12-15T17:02:04.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.713+0000 7f72af96c640 1 librados: init done 2023-12-15T17:02:04.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.714+0000 7f72af96c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:04.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.714+0000 7f72af96c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:04.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.714+0000 7f72af96c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:04.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.714+0000 7f72af96c640 1 -- 172.21.15.69:0/2577791817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a8067ac0 con 0x7f72a8142a20 2023-12-15T17:02:04.738 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.736+0000 7f729dffb640 1 -- 172.21.15.69:0/2577791817 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f72a0003970 con 0x7f72a8142a20 2023-12-15T17:02:04.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.737+0000 7f729dffb640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f728c06cba0 0x7f728c06f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.737+0000 7f72ad708640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f728c06cba0 0x7f728c06f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.737+0000 7f72ad708640 10 monclient: get_auth_request con 0x7f728c06cba0 auth_method 0 2023-12-15T17:02:04.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.738+0000 7f729dffb640 10 client.4344.objecter ms_handle_connect 0x7f728c06cba0 2023-12-15T17:02:04.739 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.738+0000 7f72ad708640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f728c06cba0 0x7f728c06f060 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72a4036000 tx=0x7f72a4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:04.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.753+0000 7f83c077c640 10 monclient: get_monmap_and_config 2023-12-15T17:02:04.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.753+0000 7f83c077c640 10 monclient: build_initial_monmap 2023-12-15T17:02:04.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.753+0000 7f83c077c640 10 monclient: monmap: 2023-12-15T17:02:04.755 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:04.755 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.755 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:04.755391+0000 2023-12-15T17:02:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:04.755391+0000 2023-12-15T17:02:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:04.756 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.754+0000 7f83c077c640 1 Processor -- start 2023-12-15T17:02:04.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.754+0000 7f83c077c640 1 -- start start 2023-12-15T17:02:04.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.754+0000 7f83c077c640 10 monclient: init 2023-12-15T17:02:04.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:04.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-12-15T17:02:04.757 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f83b80a8910 0x7f83b813ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 10 monclient(hunting): picked mon.noname-a con 0x7f83b80a8910 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a9b70 0x7f83b813f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 10 monclient(hunting): picked mon.noname-b con 0x7f83b80a9b70 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f83b8141f90 0x7f83b8142400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 10 monclient(hunting): picked mon.noname-c con 0x7f83b8141f90 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b80a8cf0 con 0x7f83b80a8910 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b813f6e0 con 0x7f83b8141f90 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80aabd0 con 0x7f83b80a9b70 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.755+0000 7f83c077c640 10 monclient(hunting): authenticate will time out at 1429.071676s 2023-12-15T17:02:04.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.771+0000 7f83bdd17640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a9b70 0x7f83b813f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.772+0000 7f83bdd17640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a9b70 0x7f83b813f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53062/0 (socket says 172.21.15.69:53062) 2023-12-15T17:02:04.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.772+0000 7f83bdd17640 1 -- 172.21.15.69:0/3077984222 learned_addr learned my addr 172.21.15.69:0/3077984222 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:04.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.772+0000 7f83bdd17640 10 monclient(hunting): get_auth_request con 0x7f83b80a9b70 auth_method 0 2023-12-15T17:02:04.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.772+0000 7f83bdd17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.772+0000 7f83bdd17640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.772+0000 7f83bdd17640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.772+0000 7f83bdd17640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.772+0000 7f83bdd17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.772+0000 7f83bdd17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.774+0000 7f83bdd17640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2023-12-15T17:02:04.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.774+0000 7f83bdd17640 1 -- 172.21.15.69:0/3077984222 >> v2:172.21.15.192:3300/0 conn(0x7f83b8141f90 msgr2=0x7f83b8142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:04.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.774+0000 7f83bdd17640 1 --2- 172.21.15.69:0/3077984222 >> v2:172.21.15.192:3300/0 conn(0x7f83b8141f90 0x7f83b8142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.774+0000 7f83bdd17640 1 -- 172.21.15.69:0/3077984222 >> v2:172.21.15.69:3300/0 conn(0x7f83b80a8910 msgr2=0x7f83b813ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:04.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.774+0000 7f83bdd17640 1 --2- 172.21.15.69:0/3077984222 >> v2:172.21.15.69:3300/0 conn(0x7f83b80a8910 0x7f83b813ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.774+0000 7f83bdd17640 10 monclient: _finish_hunting 0 2023-12-15T17:02:04.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.774+0000 7f83bdd17640 1 monclient: found mon.noname-b 2023-12-15T17:02:04.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.774+0000 7f83bdd17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:04.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.774+0000 7f83bdd17640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:04.776 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.774+0000 7f83bdd17640 1 -- 172.21.15.69:0/3077984222 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b80aaeb0 con 0x7f83b80a9b70 2023-12-15T17:02:04.777 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.776+0000 7f83bdd17640 1 --2- 172.21.15.69:0/3077984222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a9b70 0x7f83b813f1a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f83b80a92b0 tx=0x7f83b402b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea990c6e219da492 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:04.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 1 -- 172.21.15.69:0/3077984222 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4031870 con 0x7f83b80a9b70 2023-12-15T17:02:04.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:04.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 10 monclient: dump: 2023-12-15T17:02:04.781 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.781 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.781 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.782 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.782 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.782 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.782 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.782 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.782 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.782 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 10 monclient: _finish_auth 0 2023-12-15T17:02:04.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 10 monclient: _check_auth_tickets 2023-12-15T17:02:04.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:04.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 1 -- 172.21.15.69:0/3077984222 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b40319d0 con 0x7f83b80a9b70 2023-12-15T17:02:04.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:04.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 1 -- 172.21.15.69:0/3077984222 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4031d60 con 0x7f83b80a9b70 2023-12-15T17:02:04.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83c077c640 5 monclient: authenticate success, global_id 4316 2023-12-15T17:02:04.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83c077c640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:04.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83c077c640 4 set_mon_vals no callback set 2023-12-15T17:02:04.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:04.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.779+0000 7f83bd516640 10 monclient: dump: 2023-12-15T17:02:04.783 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.784 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.784 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.784 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.784 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.784 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.784 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.784 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.784 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.784 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.780+0000 7f83c077c640 10 monclient: shutdown 2023-12-15T17:02:04.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.780+0000 7f83c077c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:04.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.780+0000 7f83c077c640 1 -- 172.21.15.69:0/3077984222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a9b70 msgr2=0x7f83b813f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.780+0000 7f83c077c640 1 --2- 172.21.15.69:0/3077984222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a9b70 0x7f83b813f1a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f83b80a92b0 tx=0x7f83b402b910 comp rx=0 tx=0).stop 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.780+0000 7f83c077c640 1 -- 172.21.15.69:0/3077984222 shutdown_connections 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.780+0000 7f83c077c640 1 --2- 172.21.15.69:0/3077984222 >> v2:172.21.15.192:3300/0 conn(0x7f83b8141f90 0x7f83b8142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.780+0000 7f83c077c640 1 --2- 172.21.15.69:0/3077984222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a9b70 0x7f83b813f1a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.780+0000 7f83c077c640 1 --2- 172.21.15.69:0/3077984222 >> v2:172.21.15.69:3300/0 conn(0x7f83b80a8910 0x7f83b813ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.780+0000 7f83c077c640 1 -- 172.21.15.69:0/3077984222 >> 172.21.15.69:0/3077984222 conn(0x7f83b80b7780 msgr2=0x7f83b80b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.780+0000 7f83c077c640 1 -- 172.21.15.69:0/3077984222 shutdown_connections 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.780+0000 7f83c077c640 1 -- 172.21.15.69:0/3077984222 wait complete. 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.781+0000 7f83c077c640 10 monclient: build_initial_monmap 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.781+0000 7f83c077c640 10 monclient: monmap: 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:04.787 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.781+0000 7f83c077c640 1 librados: starting msgr at 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.781+0000 7f83c077c640 1 librados: starting objecter 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.781+0000 7f83c077c640 1 Processor -- start 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.781+0000 7f83c077c640 1 -- start start 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.781+0000 7f83c077c640 1 librados: setting wanted keys 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.781+0000 7f83c077c640 1 librados: calling monclient init 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.781+0000 7f83c077c640 10 monclient: init 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:02:04.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a8910 0x7f83b81c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 10 monclient(hunting): picked mon.noname-c con 0x7f83b80a8910 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83b8141f90 0x7f83b813ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 10 monclient(hunting): picked mon.noname-b con 0x7f83b8141f90 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b813b0d0 0x7f83b81d72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 10 monclient(hunting): picked mon.noname-a con 0x7f83b813b0d0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b81c8f20 con 0x7f83b813b0d0 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b81c9cc0 con 0x7f83b8141f90 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b81c9fa0 con 0x7f83b80a8910 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83c077c640 10 monclient(hunting): authenticate will time out at 1429.098957s 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.782+0000 7f83be518640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a8910 0x7f83b81c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.783+0000 7f83be518640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a8910 0x7f83b81c86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53076/0 (socket says 172.21.15.69:53076) 2023-12-15T17:02:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.783+0000 7f83be518640 1 -- 172.21.15.69:0/1092473715 learned_addr learned my addr 172.21.15.69:0/1092473715 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.783+0000 7f83be518640 10 monclient(hunting): get_auth_request con 0x7f83b80a8910 auth_method 0 2023-12-15T17:02:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.783+0000 7f83be518640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.783+0000 7f83be518640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.783+0000 7f83be518640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.784+0000 7f83be518640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.784+0000 7f83be518640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.784+0000 7f83be518640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.787+0000 7f83be518640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2023-12-15T17:02:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.787+0000 7f83be518640 1 -- 172.21.15.69:0/1092473715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83b8141f90 msgr2=0x7f83b813ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.787+0000 7f83be518640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83b8141f90 0x7f83b813ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.787+0000 7f83be518640 1 -- 172.21.15.69:0/1092473715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b813b0d0 msgr2=0x7f83b81d72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.787+0000 7f83be518640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b813b0d0 0x7f83b81d72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.787+0000 7f83be518640 10 monclient: _finish_hunting 0 2023-12-15T17:02:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.787+0000 7f83be518640 1 monclient: found mon.noname-c 2023-12-15T17:02:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.787+0000 7f83be518640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.787+0000 7f83be518640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:02:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.787+0000 7f83be518640 1 -- 172.21.15.69:0/1092473715 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b40094d0 con 0x7f83b80a8910 2023-12-15T17:02:04.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.789+0000 7f83be518640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a8910 0x7f83b81c86e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f83a800b410 tx=0x7f83a800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:04.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.789+0000 7f83aeffd640 10 client.?.objecter ms_handle_connect 0x7f83b80a8910 2023-12-15T17:02:04.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.789+0000 7f83aeffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 1 -- 172.21.15.69:0/1092473715 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8011830 con 0x7f83b80a8910 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 10 monclient: dump: 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 1 -- 172.21.15.69:0/1092473715 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a80119b0 con 0x7f83b80a8910 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 1 -- 172.21.15.69:0/1092473715 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8011d60 con 0x7f83b80a8910 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83c077c640 5 monclient: authenticate success, global_id 4319 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83aeffd640 10 monclient: dump: 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.796 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83c077c640 10 monclient: _renew_subs 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83c077c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83bcd15640 4 set_mon_vals no callback set 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83c077c640 1 -- 172.21.15.69:0/1092473715 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b80b06a0 con 0x7f83b80a8910 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83c077c640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83c077c640 10 monclient: _renew_subs 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83c077c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:04.797 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83c077c640 1 -- 172.21.15.69:0/1092473715 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b80b1610 con 0x7f83b80a8910 2023-12-15T17:02:04.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.793+0000 7f83c077c640 1 librados: init done 2023-12-15T17:02:04.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.794+0000 7f83c077c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:04.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.794+0000 7f83c077c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:04.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.794+0000 7f83c077c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:04.798 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.794+0000 7f83c077c640 1 -- 172.21.15.69:0/1092473715 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b8075b60 con 0x7f83b80a8910 2023-12-15T17:02:04.812 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-15T17:02:04.845 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-15T17:02:04.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.902+0000 7fcbabfff640 1 -- 172.21.15.69:0/861125557 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fcbb8013470 con 0x7fcbc40a8e10 2023-12-15T17:02:04.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.902+0000 7fcbabfff640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcbb006cb50 0x7fcbb006f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.903+0000 7fcbc2ffd640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcbb006cb50 0x7fcbb006f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.903+0000 7fcbc2ffd640 10 monclient: get_auth_request con 0x7fcbb006cb50 auth_method 0 2023-12-15T17:02:04.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.903+0000 7fcbabfff640 10 client.4357.objecter ms_handle_connect 0x7fcbb006cb50 2023-12-15T17:02:04.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.903+0000 7fcbc2ffd640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcbb006cb50 0x7fcbb006f010 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4036000 tx=0x7fcbb403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:04.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.905+0000 7fcbabfff640 1 -- 172.21.15.69:0/861125557 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcbb8058e10 con 0x7fcbc40a8e10 2023-12-15T17:02:04.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.905+0000 7fcbabfff640 10 client.4357.objecter ms_dispatch 0x7fcbc4000b90 osd_map(23..23 src has 1..23) v4 2023-12-15T17:02:04.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.905+0000 7fcbabfff640 3 client.4357.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:02:04.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.905+0000 7fcbabfff640 3 client.4357.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:02:04.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.905+0000 7fcbabfff640 20 client.4357.objecter dump_active .. 0 homeless 2023-12-15T17:02:04.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.911+0000 7f2b0f997640 10 monclient: get_monmap_and_config 2023-12-15T17:02:04.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.911+0000 7f2b0f997640 10 monclient: build_initial_monmap 2023-12-15T17:02:04.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.911+0000 7f2b0f997640 10 monclient: monmap: 2023-12-15T17:02:04.913 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:04.914 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.914 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:04.913351+0000 2023-12-15T17:02:04.914 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:04.913351+0000 2023-12-15T17:02:04.914 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:04.914 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.914 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:04.914 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:04.914 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:04.914 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:04.914 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:04.914 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.912+0000 7f2b0f997640 1 Processor -- start 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.912+0000 7f2b0f997640 1 -- start start 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.912+0000 7f2b0f997640 10 monclient: init 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2b080a8a30 0x7f2b081424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 10 monclient(hunting): picked mon.noname-a con 0x7f2b080a8a30 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b080a8e10 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08142a20 0x7f2b08145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 10 monclient(hunting): picked mon.noname-b con 0x7f2b08142a20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b08142eb0 con 0x7f2b080a8e10 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b080aaa10 con 0x7f2b080a8a30 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b080aac10 con 0x7f2b08142a20 2023-12-15T17:02:04.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:04.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.913+0000 7f2b0f997640 10 monclient(hunting): authenticate will time out at 1429.229708s 2023-12-15T17:02:04.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.914+0000 7fcbabfff640 1 -- 172.21.15.69:0/861125557 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fcbb8055150 con 0x7fcbc40a8e10 2023-12-15T17:02:04.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.914+0000 7fcbabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:04.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.914+0000 7fcbabfff640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:04.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-15T17:02:04.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.957+0000 7f9ad2f2c640 10 monclient: get_monmap_and_config 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.957+0000 7f9ad2f2c640 10 monclient: build_initial_monmap 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.957+0000 7f9ad2f2c640 10 monclient: monmap: 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:04.958657+0000 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:04.958657+0000 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.957+0000 7f9ad2f2c640 1 Processor -- start 2023-12-15T17:02:04.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 1 -- start start 2023-12-15T17:02:04.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 10 monclient: init 2023-12-15T17:02:04.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:04.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-12-15T17:02:04.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9acc0a8a30 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:04.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9acc142bf0 0x7f9acc13ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 10 monclient(hunting): picked mon.noname-c con 0x7f9acc142bf0 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:04.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9acc0a8e10 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:04.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc0aac10 con 0x7f9acc0a8e10 2023-12-15T17:02:04.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc0aaef0 con 0x7f9acc0a8a30 2023-12-15T17:02:04.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9acc0aaa10 con 0x7f9acc142bf0 2023-12-15T17:02:04.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:04.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.958+0000 7f9ad2f2c640 10 monclient(hunting): authenticate will time out at 1429.275133s 2023-12-15T17:02:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.968+0000 7f2b0cf32640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f2b080a8e10 0x7f2b081454a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:60608/0 (socket says 172.21.15.69:60608) 2023-12-15T17:02:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.968+0000 7f2b0cf32640 1 -- 172.21.15.69:0/233696706 learned_addr learned my addr 172.21.15.69:0/233696706 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:04.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.968+0000 7f9acbfff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9acc142bf0 0x7f9acc13ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.969+0000 7f9acbfff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9acc142bf0 0x7f9acc13ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37422/0 (socket says 172.21.15.69:37422) 2023-12-15T17:02:04.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.969+0000 7f9acbfff640 1 -- 172.21.15.69:0/2761106536 learned_addr learned my addr 172.21.15.69:0/2761106536 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:04.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.969+0000 7f9acbfff640 10 monclient(hunting): get_auth_request con 0x7f9acc142bf0 auth_method 0 2023-12-15T17:02:04.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.969+0000 7f9acbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.969+0000 7f9acbfff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.969+0000 7f9acbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.970+0000 7f9acbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.970+0000 7f9acbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.970+0000 7f9acbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.971+0000 7f9acbfff640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2023-12-15T17:02:04.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.971+0000 7f9acbfff640 1 -- 172.21.15.69:0/2761106536 >> v1:172.21.15.192:6789/0 conn(0x7f9acc0a8a30 legacy=0x7f9acc1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.971+0000 7f9acbfff640 1 -- 172.21.15.69:0/2761106536 >> v1:172.21.15.69:6789/0 conn(0x7f9acc0a8e10 legacy=0x7f9acc13f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:04.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.971+0000 7f9acbfff640 10 monclient: _finish_hunting 0 2023-12-15T17:02:04.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.971+0000 7f9acbfff640 1 monclient: found mon.noname-c 2023-12-15T17:02:04.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.971+0000 7f9acbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:04.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.971+0000 7f9acbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:02:04.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.971+0000 7f9acbfff640 1 -- 172.21.15.69:0/2761106536 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc0a97f0 con 0x7f9acc142bf0 2023-12-15T17:02:04.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.977+0000 7f9acbfff640 1 --2- 172.21.15.69:0/2761106536 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9acc142bf0 0x7f9acc13ecf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009870 tx=0x7f9ab802ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f812a91a41f141c2 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:04.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.977+0000 7f83aeffd640 1 -- 172.21.15.69:0/1092473715 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f83a801a400 con 0x7f83b80a8910 2023-12-15T17:02:04.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.977+0000 7f83aeffd640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f839c06cba0 0x7f839c06f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.978+0000 7f83bdd17640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f839c06cba0 0x7f839c06f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.978+0000 7f83bdd17640 10 monclient: get_auth_request con 0x7f839c06cba0 auth_method 0 2023-12-15T17:02:04.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.978+0000 7f83aeffd640 10 client.4319.objecter ms_handle_connect 0x7f839c06cba0 2023-12-15T17:02:04.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.978+0000 7f83bdd17640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f839c06cba0 0x7f839c06f060 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f83b4036000 tx=0x7f83b4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 1 -- 172.21.15.69:0/2761106536 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab802f870 con 0x7f9acc142bf0 2023-12-15T17:02:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 10 monclient: dump: 2023-12-15T17:02:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.982 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 10 monclient: _finish_auth 0 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 10 monclient: _check_auth_tickets 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 1 -- 172.21.15.69:0/2761106536 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab802f9d0 con 0x7f9acc142bf0 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 1 -- 172.21.15.69:0/2761106536 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab802fda0 con 0x7f9acc142bf0 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9acb7fe640 10 monclient: dump: 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:04.983 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:04.984 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9ad2f2c640 5 monclient: authenticate success, global_id 4363 2023-12-15T17:02:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9ad2f2c640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:04.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.980+0000 7f9ad2f2c640 4 set_mon_vals no callback set 2023-12-15T17:02:04.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.981+0000 7f9ad2f2c640 10 monclient: shutdown 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.981+0000 7f9ad2f2c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.981+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/2761106536 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9acc142bf0 msgr2=0x7f9acc13ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.981+0000 7f9ad2f2c640 1 --2- 172.21.15.69:0/2761106536 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9acc142bf0 0x7f9acc13ecf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009870 tx=0x7f9ab802ba50 comp rx=0 tx=0).stop 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.981+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/2761106536 shutdown_connections 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.981+0000 7f9ad2f2c640 1 --2- 172.21.15.69:0/2761106536 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9acc142bf0 0x7f9acc13ecf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.981+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/2761106536 >> 172.21.15.69:0/2761106536 conn(0x7f9acc061a30 msgr2=0x7f9acc0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.981+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/2761106536 shutdown_connections 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.981+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/2761106536 wait complete. 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.982+0000 7f9ad2f2c640 10 monclient: build_initial_monmap 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.982+0000 7f9ad2f2c640 10 monclient: monmap: 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:04.986 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:04.987 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:04.987 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:04.987 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:04.989 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:04.989 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:04.989 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:04.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.982+0000 7f9ad2f2c640 1 librados: starting msgr at 2023-12-15T17:02:04.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.982+0000 7f9ad2f2c640 1 librados: starting objecter 2023-12-15T17:02:04.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.982+0000 7f9ad2f2c640 1 Processor -- start 2023-12-15T17:02:04.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.984+0000 7f9ad2f2c640 1 -- start start 2023-12-15T17:02:04.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.984+0000 7f9ad2f2c640 1 librados: setting wanted keys 2023-12-15T17:02:04.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.984+0000 7f9ad2f2c640 1 librados: calling monclient init 2023-12-15T17:02:04.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.984+0000 7f9ad2f2c640 10 monclient: init 2023-12-15T17:02:04.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:04.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:02:04.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9acc0a8a30 0x7f9acc06da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 10 monclient(hunting): picked mon.noname-b con 0x7f9acc0a8a30 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:04.990 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9acc0a8e10 0x7f9acc1de960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 10 monclient(hunting): picked mon.noname-c con 0x7f9acc0a8e10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9acc142bf0 0x7f9acc1df150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 10 monclient(hunting): picked mon.noname-a con 0x7f9acc142bf0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc071600 con 0x7f9acc142bf0 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc0abfc0 con 0x7f9acc0a8a30 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc06f6d0 con 0x7f9acc0a8e10 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.985+0000 7f9ad2f2c640 10 monclient(hunting): authenticate will time out at 1429.302182s 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.986+0000 7f9ad0cc8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9acc0a8a30 0x7f9acc06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.986+0000 7f9ad0cc8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9acc0a8a30 0x7f9acc06da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37430/0 (socket says 172.21.15.69:37430) 2023-12-15T17:02:04.991 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.986+0000 7f9ad0cc8640 1 -- 172.21.15.69:0/3086865092 learned_addr learned my addr 172.21.15.69:0/3086865092 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:04.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.986+0000 7f9ad0cc8640 10 monclient(hunting): get_auth_request con 0x7f9acc0a8a30 auth_method 0 2023-12-15T17:02:04.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.986+0000 7f9ad0cc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:04.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.986+0000 7f9ad0cc8640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:04.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.986+0000 7f9ad0cc8640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:04.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.987+0000 7f9ad0cc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:04.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.987+0000 7f9ad0cc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:04.992 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.987+0000 7f9ad0cc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:04.992 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-15T17:02:04.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.990+0000 7f9ad0cc8640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2023-12-15T17:02:04.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.990+0000 7f9ad0cc8640 1 -- 172.21.15.69:0/3086865092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9acc0a8e10 msgr2=0x7f9acc1de960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.990+0000 7f9ad0cc8640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9acc0a8e10 0x7f9acc1de960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.990+0000 7f9ad0cc8640 1 -- 172.21.15.69:0/3086865092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9acc142bf0 msgr2=0x7f9acc1df150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:04.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.990+0000 7f9ad0cc8640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9acc142bf0 0x7f9acc1df150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:04.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.990+0000 7f9ad0cc8640 10 monclient: _finish_hunting 0 2023-12-15T17:02:04.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.990+0000 7f9ad0cc8640 1 monclient: found mon.noname-b 2023-12-15T17:02:04.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.990+0000 7f9ad0cc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:04.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.990+0000 7f9ad0cc8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:02:04.994 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.990+0000 7f9ad0cc8640 1 -- 172.21.15.69:0/3086865092 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab80094d0 con 0x7f9acc0a8a30 2023-12-15T17:02:04.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.993+0000 7f9ad0cc8640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9acc0a8a30 0x7f9acc06da90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9ac000c450 tx=0x7f9ac000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:04.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.993+0000 7f9ac97fa640 10 client.?.objecter ms_handle_connect 0x7f9acc0a8a30 2023-12-15T17:02:04.995 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.993+0000 7f9ac97fa640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:05.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 1 -- 172.21.15.69:0/3086865092 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac00052d0 con 0x7f9acc0a8a30 2023-12-15T17:02:05.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:05.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 10 monclient: dump: 2023-12-15T17:02:05.000 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.000 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.000 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.000 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.000 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.000 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.000 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 10 monclient: _finish_auth 0 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 10 monclient: _check_auth_tickets 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 1 -- 172.21.15.69:0/3086865092 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac0005430 con 0x7f9acc0a8a30 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 1 -- 172.21.15.69:0/3086865092 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0012870 con 0x7f9acc0a8a30 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ac97fa640 10 monclient: dump: 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.001 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9acaffd640 4 set_mon_vals no callback set 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.997+0000 7f9ad2f2c640 5 monclient: authenticate success, global_id 4366 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.998+0000 7f9ad2f2c640 10 monclient: _renew_subs 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.998+0000 7f9ad2f2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.998+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/3086865092 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9acc1def00 con 0x7f9acc0a8a30 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.998+0000 7f9ad2f2c640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.998+0000 7f9ad2f2c640 10 monclient: _renew_subs 2023-12-15T17:02:05.002 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.998+0000 7f9ad2f2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:05.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.998+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/3086865092 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9acc13af10 con 0x7f9acc0a8a30 2023-12-15T17:02:05.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.998+0000 7f9ad2f2c640 1 librados: init done 2023-12-15T17:02:05.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.999+0000 7f9ad2f2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.999+0000 7f9ad2f2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.999+0000 7f9ad2f2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:05.003 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:04.999+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/3086865092 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9acc067ac0 con 0x7f9acc0a8a30 2023-12-15T17:02:05.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.026+0000 7f729dffb640 1 -- 172.21.15.69:0/2577791817 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f72a00583f0 con 0x7f72a8142a20 2023-12-15T17:02:05.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.026+0000 7f729dffb640 10 client.4344.objecter ms_dispatch 0x7f72a8000b90 osd_map(23..23 src has 1..23) v4 2023-12-15T17:02:05.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.026+0000 7f729dffb640 3 client.4344.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:02:05.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.026+0000 7f729dffb640 3 client.4344.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:02:05.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.026+0000 7f729dffb640 20 client.4344.objecter dump_active .. 0 homeless 2023-12-15T17:02:05.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.026+0000 7f729dffb640 1 -- 172.21.15.69:0/2577791817 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f72a0054730 con 0x7f72a8142a20 2023-12-15T17:02:05.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.026+0000 7f729dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.026+0000 7f729dffb640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:05.032 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.031+0000 7f2b0d733640 1 --2- 172.21.15.69:0/233696706 >> v2:172.21.15.69:3300/0 conn(0x7f2b080a8a30 0x7f2b081424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.031+0000 7f2b0d733640 10 monclient(hunting): get_auth_request con 0x7f2b080a8a30 auth_method 0 2023-12-15T17:02:05.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.031+0000 7f2b0d733640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:05.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.031+0000 7f2b0d733640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:05.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.031+0000 7f2b0d733640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:05.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.032+0000 7f2b0d733640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:05.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.032+0000 7f2b0d733640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:05.033 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.032+0000 7f2b0d733640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.033+0000 7f2b0d733640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2023-12-15T17:02:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.033+0000 7f2b0d733640 1 -- 172.21.15.69:0/233696706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08142a20 msgr2=0x7f2b08145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.033+0000 7f2b0d733640 1 --2- 172.21.15.69:0/233696706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08142a20 0x7f2b08145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.033+0000 7f2b0d733640 1 -- 172.21.15.69:0/233696706 >> v1:172.21.15.192:6789/0 conn(0x7f2b080a8e10 legacy=0x7f2b081454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.034+0000 7f2b0d733640 10 monclient: _finish_hunting 0 2023-12-15T17:02:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.034+0000 7f2b0d733640 1 monclient: found mon.noname-a 2023-12-15T17:02:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.034+0000 7f2b0d733640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.034+0000 7f2b0d733640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.034+0000 7f2b0d733640 1 -- 172.21.15.69:0/233696706 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b080aaef0 con 0x7f2b080a8a30 2023-12-15T17:02:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.037+0000 7f2b0d733640 1 --2- 172.21.15.69:0/233696706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b080a8a30 0x7f2b081424e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2b040097a0 tx=0x7f2b0402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed41f5a155cd159 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.051+0000 7f8098d48640 10 monclient: get_monmap_and_config 2023-12-15T17:02:05.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.051+0000 7f8098d48640 10 monclient: build_initial_monmap 2023-12-15T17:02:05.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.051+0000 7f8098d48640 10 monclient: monmap: 2023-12-15T17:02:05.053 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:05.053 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.053 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:05.052965+0000 2023-12-15T17:02:05.053 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:05.052965+0000 2023-12-15T17:02:05.053 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:05.053 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.054 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:05.054 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:05.054 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:05.054 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:05.054 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:05.054 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f2afffff640 1 -- 172.21.15.69:0/233696706 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04030610 con 0x7f2b080a8a30 2023-12-15T17:02:05.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f2afffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.054 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f2afffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f2afffff640 10 monclient: dump: 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f2afffff640 10 monclient: _finish_auth 0 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f2afffff640 10 monclient: _check_auth_tickets 2023-12-15T17:02:05.055 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f2afffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:05.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f2b0f997640 5 monclient: authenticate success, global_id 4347 2023-12-15T17:02:05.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f2b0f997640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:05.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f8098d48640 1 Processor -- start 2023-12-15T17:02:05.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f8098d48640 1 -- start start 2023-12-15T17:02:05.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.052+0000 7f8098d48640 10 monclient: init 2023-12-15T17:02:05.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:05.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-12-15T17:02:05.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80940a8a30 0x7f80941424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 10 monclient(hunting): picked mon.noname-b con 0x7f80940a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:05.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f80940a8e10 0x7f8094142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 10 monclient(hunting): picked mon.noname-c con 0x7f80940a8e10 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:05.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80941454a0 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:05.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80940aac10 con 0x7f80941454a0 2023-12-15T17:02:05.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80940aaa10 con 0x7f80940a8e10 2023-12-15T17:02:05.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80940aaef0 con 0x7f80940a8a30 2023-12-15T17:02:05.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:05.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.053+0000 7f8098d48640 10 monclient(hunting): authenticate will time out at 1429.370040s 2023-12-15T17:02:05.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-15T17:02:05.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.063+0000 7f83aeffd640 1 -- 172.21.15.69:0/1092473715 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f83a8059280 con 0x7f83b80a8910 2023-12-15T17:02:05.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.063+0000 7f83aeffd640 10 client.4319.objecter ms_dispatch 0x7f83b8000b90 osd_map(23..23 src has 1..23) v4 2023-12-15T17:02:05.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.063+0000 7f83aeffd640 3 client.4319.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:02:05.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.063+0000 7f83aeffd640 3 client.4319.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:02:05.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.063+0000 7f83aeffd640 20 client.4319.objecter dump_active .. 0 homeless 2023-12-15T17:02:05.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.070+0000 7f8092d9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f80941454a0 0x7f8094145930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59756/0 (socket says 172.21.15.69:59756) 2023-12-15T17:02:05.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.070+0000 7f8092d9d640 1 -- 172.21.15.69:0/1774178786 learned_addr learned my addr 172.21.15.69:0/1774178786 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:05.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.122+0000 7f9dae661640 10 monclient: get_monmap_and_config 2023-12-15T17:02:05.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.122+0000 7f9dae661640 10 monclient: build_initial_monmap 2023-12-15T17:02:05.124 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.122+0000 7f9dae661640 10 monclient: monmap: 2023-12-15T17:02:05.124 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:05.125 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.125 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:05.124249+0000 2023-12-15T17:02:05.125 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:05.124249+0000 2023-12-15T17:02:05.125 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:05.125 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.125 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:05.125 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:05.125 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:05.125 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:05.125 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:05.125 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.123+0000 7f9dae661640 1 Processor -- start 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.123+0000 7f9dae661640 1 -- start start 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.123+0000 7f9dae661640 10 monclient: init 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8a30 0x7f9da81424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 10 monclient(hunting): picked mon.noname-b con 0x7f9da80a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9da80a8e10 0x7f9da8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 10 monclient(hunting): picked mon.noname-a con 0x7f9da80a8e10 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9da81454a0 0x7f9da8145930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 10 monclient(hunting): picked mon.noname-c con 0x7f9da81454a0 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:05.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da80aaa10 con 0x7f9da80a8e10 2023-12-15T17:02:05.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da80aac10 con 0x7f9da81454a0 2023-12-15T17:02:05.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80aaef0 con 0x7f9da80a8a30 2023-12-15T17:02:05.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:05.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.124+0000 7f9dae661640 10 monclient(hunting): authenticate will time out at 1429.441029s 2023-12-15T17:02:05.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.133+0000 7f83aeffd640 1 -- 172.21.15.69:0/1092473715 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f83a8054ec0 con 0x7f83b80a8910 2023-12-15T17:02:05.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.133+0000 7f83aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.133+0000 7f83aeffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:05.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.134+0000 7f809259c640 1 --2- 172.21.15.69:0/1774178786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80940a8a30 0x7f80941424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.139+0000 7f809259c640 10 monclient(hunting): get_auth_request con 0x7f80940a8a30 auth_method 0 2023-12-15T17:02:05.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.139+0000 7f809259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:05.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.139+0000 7f809259c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:05.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.139+0000 7f809259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:05.143 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.142+0000 7f809259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:05.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.142+0000 7f809259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:05.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.142+0000 7f809259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.145+0000 7f9da7fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8a30 0x7f9da81424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.144+0000 7f809259c640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2023-12-15T17:02:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.144+0000 7f809259c640 1 -- 172.21.15.69:0/1774178786 >> v2:172.21.15.192:3300/0 conn(0x7f80940a8e10 msgr2=0x7f8094142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.144+0000 7f809259c640 1 --2- 172.21.15.69:0/1774178786 >> v2:172.21.15.192:3300/0 conn(0x7f80940a8e10 0x7f8094142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.144+0000 7f809259c640 1 -- 172.21.15.69:0/1774178786 >> v1:172.21.15.69:6789/0 conn(0x7f80941454a0 legacy=0x7f8094145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.144+0000 7f809259c640 10 monclient: _finish_hunting 0 2023-12-15T17:02:05.147 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.144+0000 7f809259c640 1 monclient: found mon.noname-b 2023-12-15T17:02:05.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.145+0000 7f809259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:05.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.145+0000 7f809259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.148 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.145+0000 7f809259c640 1 -- 172.21.15.69:0/1774178786 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80940a97f0 con 0x7f80940a8a30 2023-12-15T17:02:05.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.147+0000 7f9da7fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8a30 0x7f9da81424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53094/0 (socket says 172.21.15.69:53094) 2023-12-15T17:02:05.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.147+0000 7f9da7fff640 1 -- 172.21.15.69:0/1142871463 learned_addr learned my addr 172.21.15.69:0/1142871463 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:05.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.147+0000 7f9da7fff640 10 monclient(hunting): get_auth_request con 0x7f9da80a8a30 auth_method 0 2023-12-15T17:02:05.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.147+0000 7f9da7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:05.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.147+0000 7f9da7fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:05.149 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.147+0000 7f9da7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:05.150 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.149+0000 7f809259c640 1 --2- 172.21.15.69:0/1774178786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80940a8a30 0x7f80941424e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f80840097d0 tx=0x7f808402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0888f8e08f27f07 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.149+0000 7f9da7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:05.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.149+0000 7f9da7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:05.151 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.149+0000 7f9da7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.151+0000 7f9da7fff640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2023-12-15T17:02:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.151+0000 7f9da7fff640 1 -- 172.21.15.69:0/1142871463 >> v2:172.21.15.192:3300/0 conn(0x7f9da81454a0 msgr2=0x7f9da8145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.151+0000 7f9da7fff640 1 --2- 172.21.15.69:0/1142871463 >> v2:172.21.15.192:3300/0 conn(0x7f9da81454a0 0x7f9da8145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.151+0000 7f9da7fff640 1 -- 172.21.15.69:0/1142871463 >> v2:172.21.15.69:3300/0 conn(0x7f9da80a8e10 msgr2=0x7f9da8142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.151+0000 7f9da7fff640 1 --2- 172.21.15.69:0/1142871463 >> v2:172.21.15.69:3300/0 conn(0x7f9da80a8e10 0x7f9da8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.151+0000 7f9da7fff640 10 monclient: _finish_hunting 0 2023-12-15T17:02:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.151+0000 7f9da7fff640 1 monclient: found mon.noname-b 2023-12-15T17:02:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.151+0000 7f9da7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:05.153 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.151+0000 7f9da7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.154 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.151+0000 7f9da7fff640 1 -- 172.21.15.69:0/1142871463 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da80a97f0 con 0x7f9da80a8a30 2023-12-15T17:02:05.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.153+0000 7f2afffff640 1 -- 172.21.15.69:0/233696706 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b04030770 con 0x7f2b080a8a30 2023-12-15T17:02:05.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.153+0000 7f2afffff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:05.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.153+0000 7f2b0f997640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:05.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.153+0000 7f2b0f997640 4 set_mon_vals no callback set 2023-12-15T17:02:05.155 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.153+0000 7f2afffff640 1 -- 172.21.15.69:0/233696706 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04030610 con 0x7f2b080a8a30 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.153+0000 7f2afffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.153+0000 7f2afffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.153+0000 7f2afffff640 10 monclient: dump: 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.156 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.154+0000 7f2b0f997640 10 monclient: shutdown 2023-12-15T17:02:05.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.154+0000 7f2b0f997640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:05.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.154+0000 7f2b0f997640 1 -- 172.21.15.69:0/233696706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b080a8a30 msgr2=0x7f2b081424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.154+0000 7f2b0f997640 1 --2- 172.21.15.69:0/233696706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b080a8a30 0x7f2b081424e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2b040097a0 tx=0x7f2b0402b960 comp rx=0 tx=0).stop 2023-12-15T17:02:05.158 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.154+0000 7f2b0f997640 1 -- 172.21.15.69:0/233696706 shutdown_connections 2023-12-15T17:02:05.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.155+0000 7f2b0f997640 1 --2- 172.21.15.69:0/233696706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08142a20 0x7f2b08145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.155+0000 7f2b0f997640 1 --2- 172.21.15.69:0/233696706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b080a8a30 0x7f2b081424e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.155+0000 7f2b0f997640 1 -- 172.21.15.69:0/233696706 >> 172.21.15.69:0/233696706 conn(0x7f2b08061a30 msgr2=0x7f2b080b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:05.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.155+0000 7f2b0f997640 1 -- 172.21.15.69:0/233696706 shutdown_connections 2023-12-15T17:02:05.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.155+0000 7f2b0f997640 1 -- 172.21.15.69:0/233696706 wait complete. 2023-12-15T17:02:05.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.156+0000 7f2b0f997640 10 monclient: build_initial_monmap 2023-12-15T17:02:05.159 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.156+0000 7f2b0f997640 10 monclient: monmap: 2023-12-15T17:02:05.159 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:05.159 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:05.159 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:05.159 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:05.160 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:05.160 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.160 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:05.160 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:05.160 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:05.160 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.157+0000 7f2b0f997640 1 librados: starting msgr at 2023-12-15T17:02:05.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.157+0000 7f2b0f997640 1 librados: starting objecter 2023-12-15T17:02:05.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.157+0000 7f2b0f997640 1 Processor -- start 2023-12-15T17:02:05.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.157+0000 7f2b0f997640 1 -- start start 2023-12-15T17:02:05.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.157+0000 7f2b0f997640 1 librados: setting wanted keys 2023-12-15T17:02:05.162 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.157+0000 7f2b0f997640 1 librados: calling monclient init 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.157+0000 7f2b0f997640 10 monclient: init 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b080a8a30 0x7f2b0813b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 10 monclient(hunting): picked mon.noname-c con 0x7f2b080a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b080a8e10 0x7f2b081cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 10 monclient(hunting): picked mon.noname-a con 0x7f2b080a8e10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b08142a20 0x7f2b081db390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 10 monclient(hunting): picked mon.noname-b con 0x7f2b08142a20 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b081db8d0 con 0x7f2b080a8e10 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b081daf00 con 0x7f2b08142a20 2023-12-15T17:02:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b081cdea0 con 0x7f2b080a8a30 2023-12-15T17:02:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.158+0000 7f2b0f997640 10 monclient(hunting): authenticate will time out at 1429.475092s 2023-12-15T17:02:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.159+0000 7f2b0d733640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b080a8a30 0x7f2b0813b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.159+0000 7f2b0d733640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b080a8a30 0x7f2b0813b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53102/0 (socket says 172.21.15.69:53102) 2023-12-15T17:02:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.159+0000 7f2b0d733640 1 -- 172.21.15.69:0/546692321 learned_addr learned my addr 172.21.15.69:0/546692321 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.159+0000 7f2b0d733640 10 monclient(hunting): get_auth_request con 0x7f2b080a8a30 auth_method 0 2023-12-15T17:02:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.159+0000 7f2b0d733640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.159+0000 7f2b0d733640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.159+0000 7f2b0d733640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.160+0000 7f2b0cf32640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b080a8e10 0x7f2b081cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.160+0000 7f9da7fff640 1 --2- 172.21.15.69:0/1142871463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8a30 0x7f9da81424e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009870 tx=0x7f9d9802b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6ea7de85e392786 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.160+0000 7f2b0cf32640 10 monclient(hunting): get_auth_request con 0x7f2b080a8e10 auth_method 0 2023-12-15T17:02:05.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.160+0000 7f2b0cf32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.160+0000 7f2b0cf32640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.160+0000 7f2b0cf32640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.160+0000 7f2b0d733640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.160+0000 7f2b0d733640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.160+0000 7f2b0d733640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.161+0000 7f2b0cf32640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.161+0000 7f2b0cf32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.161+0000 7f2b0cf32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:05.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.165+0000 7f2b0cf32640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2023-12-15T17:02:05.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.165+0000 7f2b0cf32640 1 -- 172.21.15.69:0/546692321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b080a8a30 msgr2=0x7f2b0813b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.165+0000 7f2b0cf32640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b080a8a30 0x7f2b0813b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.165+0000 7f2b0cf32640 1 -- 172.21.15.69:0/546692321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b08142a20 msgr2=0x7f2b081db390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.165+0000 7f2b0cf32640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b08142a20 0x7f2b081db390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.165+0000 7f2b0cf32640 10 monclient: _finish_hunting 0 2023-12-15T17:02:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.165+0000 7f2b0cf32640 1 monclient: found mon.noname-a 2023-12-15T17:02:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.165+0000 7f2b0cf32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.165+0000 7f2b0cf32640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.165+0000 7f2b0cf32640 1 -- 172.21.15.69:0/546692321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b04009400 con 0x7f2b080a8e10 2023-12-15T17:02:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.169+0000 7f2b0cf32640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b080a8e10 0x7f2b081cece0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2af800c370 tx=0x7f2af800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.169+0000 7f2afdffb640 10 client.?.objecter ms_handle_connect 0x7f2b080a8e10 2023-12-15T17:02:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.169+0000 7f2afdffb640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:05.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 1 -- 172.21.15.69:0/546692321 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af8005ce0 con 0x7f2b080a8e10 2023-12-15T17:02:05.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 10 monclient: dump: 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.177 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 10 monclient: _finish_auth 0 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 10 monclient: _check_auth_tickets 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 1 -- 172.21.15.69:0/546692321 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af8005e40 con 0x7f2b080a8e10 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 1 -- 172.21.15.69:0/546692321 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af8007310 con 0x7f2b080a8e10 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2afdffb640 10 monclient: dump: 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.179 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.179 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.179 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.179 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2aff7fe640 4 set_mon_vals no callback set 2023-12-15T17:02:05.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2b0f997640 5 monclient: authenticate success, global_id 4353 2023-12-15T17:02:05.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2b0f997640 10 monclient: _renew_subs 2023-12-15T17:02:05.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2b0f997640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:05.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2b0f997640 1 -- 172.21.15.69:0/546692321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b081da990 con 0x7f2b080a8e10 2023-12-15T17:02:05.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2b0f997640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:05.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2b0f997640 10 monclient: _renew_subs 2023-12-15T17:02:05.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2b0f997640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:05.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2b0f997640 1 -- 172.21.15.69:0/546692321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b081cc470 con 0x7f2b080a8e10 2023-12-15T17:02:05.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.175+0000 7f2b0f997640 1 librados: init done 2023-12-15T17:02:05.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.176+0000 7f2b0f997640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.176+0000 7f2b0f997640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.176+0000 7f2b0f997640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:05.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.176+0000 7f2b0f997640 1 -- 172.21.15.69:0/546692321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b08067ac0 con 0x7f2b080a8e10 2023-12-15T17:02:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.182+0000 7f6c06d4e640 10 monclient: get_monmap_and_config 2023-12-15T17:02:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.182+0000 7f6c06d4e640 10 monclient: build_initial_monmap 2023-12-15T17:02:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.183+0000 7f6c06d4e640 10 monclient: monmap: 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:05.184474+0000 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:05.184474+0000 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:05.185 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.183+0000 7f6c06d4e640 1 Processor -- start 2023-12-15T17:02:05.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.183+0000 7f6c06d4e640 1 -- start start 2023-12-15T17:02:05.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.183+0000 7f6c06d4e640 10 monclient: init 2023-12-15T17:02:05.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6c000a8a30 0x7f6c001424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 10 monclient(hunting): picked mon.noname-c con 0x7f6c000a8a30 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6c000a8e10 0x7f6c00142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 10 monclient(hunting): picked mon.noname-a con 0x7f6c000a8e10 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c001454a0 0x7f6c00145930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 10 monclient(hunting): picked mon.noname-b con 0x7f6c001454a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c000aaa10 con 0x7f6c000a8e10 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c000aac10 con 0x7f6c000a8a30 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c000aaef0 con 0x7f6c001454a0 2023-12-15T17:02:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:05.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.184+0000 7f6c06d4e640 10 monclient(hunting): authenticate will time out at 1429.501012s 2023-12-15T17:02:05.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.240+0000 7f9ac97fa640 1 -- 172.21.15.69:0/3086865092 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9ac00129d0 con 0x7f9acc0a8a30 2023-12-15T17:02:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.241+0000 7f9ac97fa640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ab006cb50 0x7f9ab006f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.241+0000 7f9acbfff640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ab006cb50 0x7f9ab006f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.241+0000 7f6c052eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c001454a0 0x7f6c00145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.241+0000 7f9acbfff640 10 monclient: get_auth_request con 0x7f9ab006cb50 auth_method 0 2023-12-15T17:02:05.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 1 -- 172.21.15.69:0/1774178786 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084031830 con 0x7f80940a8a30 2023-12-15T17:02:05.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:05.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 10 monclient: dump: 2023-12-15T17:02:05.245 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.245 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.245 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 10 monclient: _finish_auth 0 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 10 monclient: _check_auth_tickets 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 1 -- 172.21.15.69:0/1774178786 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8084031990 con 0x7f80940a8a30 2023-12-15T17:02:05.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 1 -- 172.21.15.69:0/1774178786 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084031d60 con 0x7f80940a8a30 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f809159a640 10 monclient: dump: 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.247 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.248 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f8098d48640 5 monclient: authenticate success, global_id 4322 2023-12-15T17:02:05.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f8098d48640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:05.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f8098d48640 4 set_mon_vals no callback set 2023-12-15T17:02:05.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f9acbfff640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ab006cb50 0x7f9ab006f010 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8036000 tx=0x7f9ab8039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.242+0000 7f9ac97fa640 10 client.4366.objecter ms_handle_connect 0x7f9ab006cb50 2023-12-15T17:02:05.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f8098d48640 10 monclient: shutdown 2023-12-15T17:02:05.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f8098d48640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:05.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f8098d48640 1 -- 172.21.15.69:0/1774178786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80940a8a30 msgr2=0x7f80941424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f8098d48640 1 --2- 172.21.15.69:0/1774178786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80940a8a30 0x7f80941424e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f80840097d0 tx=0x7f808402b8d0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f8098d48640 1 -- 172.21.15.69:0/1774178786 shutdown_connections 2023-12-15T17:02:05.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f8098d48640 1 --2- 172.21.15.69:0/1774178786 >> v2:172.21.15.192:3300/0 conn(0x7f80940a8e10 0x7f8094142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f8098d48640 1 --2- 172.21.15.69:0/1774178786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80940a8a30 0x7f80941424e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f8098d48640 1 -- 172.21.15.69:0/1774178786 >> 172.21.15.69:0/1774178786 conn(0x7f8094061a30 msgr2=0x7f80940b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:05.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 1 -- 172.21.15.69:0/1142871463 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d98031870 con 0x7f9da80a8a30 2023-12-15T17:02:05.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 10 monclient: dump: 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:05.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:05.252 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:05.252 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 1 -- 172.21.15.69:0/1142871463 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d980319d0 con 0x7f9da80a8a30 2023-12-15T17:02:05.252 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9dae661640 5 monclient: authenticate success, global_id 4325 2023-12-15T17:02:05.252 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9dae661640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:05.252 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:05.252 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 1 -- 172.21.15.69:0/1142871463 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d98031da0 con 0x7f9da80a8a30 2023-12-15T17:02:05.252 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.252 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:05.252 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9da6ffd640 10 monclient: dump: 2023-12-15T17:02:05.252 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.253 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.253 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.253 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.253 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.253 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.253 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.253 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.253 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.253 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.253 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9dae661640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:05.253 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.243+0000 7f9dae661640 4 set_mon_vals no callback set 2023-12-15T17:02:05.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f6c052eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c001454a0 0x7f6c00145930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53104/0 (socket says 172.21.15.69:53104) 2023-12-15T17:02:05.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f6c052eb640 1 -- 172.21.15.69:0/3515690768 learned_addr learned my addr 172.21.15.69:0/3515690768 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:05.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f6c052eb640 10 monclient(hunting): get_auth_request con 0x7f6c001454a0 auth_method 0 2023-12-15T17:02:05.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f6c052eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:05.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f6c052eb640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:05.254 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f6c052eb640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:05.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f9dae661640 10 monclient: shutdown 2023-12-15T17:02:05.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f9dae661640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:05.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f9dae661640 1 -- 172.21.15.69:0/1142871463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8a30 msgr2=0x7f9da81424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f9dae661640 1 --2- 172.21.15.69:0/1142871463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8a30 0x7f9da81424e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009870 tx=0x7f9d9802b910 comp rx=0 tx=0).stop 2023-12-15T17:02:05.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f9ac97fa640 1 -- 172.21.15.69:0/3086865092 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9ac0059170 con 0x7f9acc0a8a30 2023-12-15T17:02:05.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f9ac97fa640 10 client.4366.objecter ms_dispatch 0x7f9acc000b90 osd_map(23..23 src has 1..23) v4 2023-12-15T17:02:05.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f9ac97fa640 3 client.4366.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:02:05.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f9ac97fa640 3 client.4366.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:02:05.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f8098d48640 1 -- 172.21.15.69:0/1774178786 shutdown_connections 2023-12-15T17:02:05.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.244+0000 7f9ac97fa640 20 client.4366.objecter dump_active .. 0 homeless 2023-12-15T17:02:05.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.245+0000 7f8098d48640 1 -- 172.21.15.69:0/1774178786 wait complete. 2023-12-15T17:02:05.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.245+0000 7f9dae661640 1 -- 172.21.15.69:0/1142871463 shutdown_connections 2023-12-15T17:02:05.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.245+0000 7f9dae661640 1 --2- 172.21.15.69:0/1142871463 >> v2:172.21.15.192:3300/0 conn(0x7f9da81454a0 0x7f9da8145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.245+0000 7f9dae661640 1 --2- 172.21.15.69:0/1142871463 >> v2:172.21.15.69:3300/0 conn(0x7f9da80a8e10 0x7f9da8142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.245+0000 7f9dae661640 1 --2- 172.21.15.69:0/1142871463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8a30 0x7f9da81424e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.245+0000 7f9dae661640 1 -- 172.21.15.69:0/1142871463 >> 172.21.15.69:0/1142871463 conn(0x7f9da8061a30 msgr2=0x7f9da80b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:05.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.245+0000 7f9dae661640 1 -- 172.21.15.69:0/1142871463 shutdown_connections 2023-12-15T17:02:05.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.245+0000 7f9dae661640 1 -- 172.21.15.69:0/1142871463 wait complete. 2023-12-15T17:02:05.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.246+0000 7f8098d48640 10 monclient: build_initial_monmap 2023-12-15T17:02:05.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.246+0000 7f8098d48640 10 monclient: monmap: 2023-12-15T17:02:05.258 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:05.258 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:05.258 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:05.258 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:05.258 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:05.259 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.259 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:05.259 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:05.259 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:05.259 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.246+0000 7f8098d48640 1 librados: starting msgr at 2023-12-15T17:02:05.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.246+0000 7f8098d48640 1 librados: starting objecter 2023-12-15T17:02:05.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f8098d48640 1 Processor -- start 2023-12-15T17:02:05.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f8098d48640 1 -- start start 2023-12-15T17:02:05.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f8098d48640 1 librados: setting wanted keys 2023-12-15T17:02:05.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.246+0000 7f9dae661640 10 monclient: build_initial_monmap 2023-12-15T17:02:05.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.246+0000 7f9dae661640 10 monclient: monmap: 2023-12-15T17:02:05.260 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:05.260 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:05.260 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:05.260 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:05.260 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:05.260 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.260 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:05.260 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:05.261 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:05.261 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.246+0000 7f9dae661640 1 librados: starting msgr at 2023-12-15T17:02:05.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.246+0000 7f9dae661640 1 librados: starting objecter 2023-12-15T17:02:05.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f9dae661640 1 Processor -- start 2023-12-15T17:02:05.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f8098d48640 1 librados: calling monclient init 2023-12-15T17:02:05.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f8098d48640 10 monclient: init 2023-12-15T17:02:05.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f9dae661640 1 -- start start 2023-12-15T17:02:05.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f9dae661640 1 librados: setting wanted keys 2023-12-15T17:02:05.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f9dae661640 1 librados: calling monclient init 2023-12-15T17:02:05.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f9dae661640 10 monclient: init 2023-12-15T17:02:05.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f6c052eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:05.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f6c052eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:05.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.247+0000 7f6c052eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:05.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:05.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:02:05.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80940a8a30 0x7f80941415e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:05.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-15T17:02:05.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9da80a8a30 0x7f9da80abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 10 monclient(hunting): picked mon.noname-a con 0x7f80940a8a30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:05.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80940a8e10 0x7f809413b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 10 monclient(hunting): picked mon.noname-a con 0x7f9da80a8a30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8e10 0x7f9da80ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 10 monclient(hunting): picked mon.noname-b con 0x7f80940a8e10 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:05.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80941454a0 0x7f809413bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.266 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 10 monclient(hunting): picked mon.noname-c con 0x7f9da80a8e10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:05.266 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da81454a0 0x7f9da8138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.266 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 10 monclient(hunting): picked mon.noname-c con 0x7f80941454a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:05.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 10 monclient(hunting): picked mon.noname-b con 0x7f9da81454a0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:05.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80acf00 con 0x7f9da80a8a30 2023-12-15T17:02:05.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80ae050 con 0x7f9da81454a0 2023-12-15T17:02:05.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da8144f30 con 0x7f9da80a8e10 2023-12-15T17:02:05.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:05.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f9dae661640 10 monclient(hunting): authenticate will time out at 1429.565272s 2023-12-15T17:02:05.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80940ad800 con 0x7f80940a8a30 2023-12-15T17:02:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809413c570 con 0x7f80940a8e10 2023-12-15T17:02:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80940ae280 con 0x7f80941454a0 2023-12-15T17:02:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.248+0000 7f8098d48640 10 monclient(hunting): authenticate will time out at 1429.565339s 2023-12-15T17:02:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.251+0000 7f6c052eb640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2023-12-15T17:02:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.251+0000 7f6c052eb640 1 -- 172.21.15.69:0/3515690768 >> v2:172.21.15.192:3300/0 conn(0x7f6c000a8a30 msgr2=0x7f6c001424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.251+0000 7f6c052eb640 1 --2- 172.21.15.69:0/3515690768 >> v2:172.21.15.192:3300/0 conn(0x7f6c000a8a30 0x7f6c001424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.251+0000 7f6c052eb640 1 -- 172.21.15.69:0/3515690768 >> v2:172.21.15.69:3300/0 conn(0x7f6c000a8e10 msgr2=0x7f6c00142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.251+0000 7f6c052eb640 1 --2- 172.21.15.69:0/3515690768 >> v2:172.21.15.69:3300/0 conn(0x7f6c000a8e10 0x7f6c00142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.251+0000 7f6c052eb640 10 monclient: _finish_hunting 0 2023-12-15T17:02:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.251+0000 7f6c052eb640 1 monclient: found mon.noname-b 2023-12-15T17:02:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.251+0000 7f6c052eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.251+0000 7f6c052eb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.251+0000 7f6c052eb640 1 -- 172.21.15.69:0/3515690768 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c000a97f0 con 0x7f6c001454a0 2023-12-15T17:02:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.254+0000 7f9ac97fa640 1 -- 172.21.15.69:0/3086865092 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9ac0054db0 con 0x7f9acc0a8a30 2023-12-15T17:02:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.254+0000 7f9ac97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.254+0000 7f9ac97fa640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:05.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.254+0000 7f6c052eb640 1 --2- 172.21.15.69:0/3515690768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c001454a0 0x7f6c00145930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6c000b5470 tx=0x7f6bf802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a8f81929ccb10fc server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.256+0000 7f9da77fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8e10 0x7f9da80ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.257+0000 7f9da77fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8e10 0x7f9da80ac670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53114/0 (socket says 172.21.15.69:53114) 2023-12-15T17:02:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.257+0000 7f9da77fe640 1 -- 172.21.15.69:0/536006312 learned_addr learned my addr 172.21.15.69:0/536006312 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.257+0000 7f9da77fe640 10 monclient(hunting): get_auth_request con 0x7f9da80a8e10 auth_method 0 2023-12-15T17:02:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.257+0000 7f9da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.257+0000 7f9da77fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.257+0000 7f9da77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.259+0000 7f9da77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.259+0000 7f9da77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:05.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.259+0000 7f9da77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.260+0000 7f8092d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80941454a0 0x7f809413bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f8092d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80941454a0 0x7f809413bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53124/0 (socket says 172.21.15.69:53124) 2023-12-15T17:02:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f8092d9d640 1 -- 172.21.15.69:0/1392961829 learned_addr learned my addr 172.21.15.69:0/1392961829 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f8092d9d640 10 monclient(hunting): get_auth_request con 0x7f80941454a0 auth_method 0 2023-12-15T17:02:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f8092d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:05.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f8092d9d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:05.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f8092d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:05.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 1 -- 172.21.15.69:0/3515690768 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf802cb40 con 0x7f6c001454a0 2023-12-15T17:02:05.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 10 monclient: dump: 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.276 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 10 monclient: _finish_auth 0 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 10 monclient: _check_auth_tickets 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 1 -- 172.21.15.69:0/3515690768 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf802cca0 con 0x7f6c001454a0 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 1 -- 172.21.15.69:0/3515690768 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8034610 con 0x7f6c001454a0 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6bff7fe640 10 monclient: dump: 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.277 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.278 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6c06d4e640 5 monclient: authenticate success, global_id 4331 2023-12-15T17:02:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6c06d4e640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:05.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.261+0000 7f6c06d4e640 4 set_mon_vals no callback set 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.262+0000 7f6c06d4e640 10 monclient: shutdown 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.262+0000 7f6c06d4e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.262+0000 7f6c06d4e640 1 -- 172.21.15.69:0/3515690768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c001454a0 msgr2=0x7f6c00145930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.262+0000 7f6c06d4e640 1 --2- 172.21.15.69:0/3515690768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c001454a0 0x7f6c00145930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6c000b5470 tx=0x7f6bf802bfa0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.262+0000 7f6c06d4e640 1 -- 172.21.15.69:0/3515690768 shutdown_connections 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.262+0000 7f6c06d4e640 1 --2- 172.21.15.69:0/3515690768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c001454a0 0x7f6c00145930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.262+0000 7f6c06d4e640 1 --2- 172.21.15.69:0/3515690768 >> v2:172.21.15.69:3300/0 conn(0x7f6c000a8e10 0x7f6c00142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.262+0000 7f6c06d4e640 1 --2- 172.21.15.69:0/3515690768 >> v2:172.21.15.192:3300/0 conn(0x7f6c000a8a30 0x7f6c001424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.262+0000 7f6c06d4e640 1 -- 172.21.15.69:0/3515690768 >> 172.21.15.69:0/3515690768 conn(0x7f6c00061a30 msgr2=0x7f6c000b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.263+0000 7f6c06d4e640 1 -- 172.21.15.69:0/3515690768 shutdown_connections 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.263+0000 7f6c06d4e640 1 -- 172.21.15.69:0/3515690768 wait complete. 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.264+0000 7f6c06d4e640 10 monclient: build_initial_monmap 2023-12-15T17:02:05.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.264+0000 7f6c06d4e640 10 monclient: monmap: 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.264+0000 7f6c06d4e640 1 librados: starting msgr at 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.264+0000 7f6c06d4e640 1 librados: starting objecter 2023-12-15T17:02:05.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.265+0000 7f6c06d4e640 1 Processor -- start 2023-12-15T17:02:05.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.265+0000 7fcbc9a39640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-15T17:02:05.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.265+0000 7fcbc9a39640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-15T17:02:05.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.265+0000 7fcbc9a39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:05.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.265+0000 7fcbc9a39640 1 -- 172.21.15.69:0/861125557 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fcbc4075b60 con 0x7fcbc40a8e10 2023-12-15T17:02:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.265+0000 7f6c06d4e640 1 -- start start 2023-12-15T17:02:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.265+0000 7f6c06d4e640 1 librados: setting wanted keys 2023-12-15T17:02:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.265+0000 7f6c06d4e640 1 librados: calling monclient init 2023-12-15T17:02:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.265+0000 7f6c06d4e640 10 monclient: init 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.266+0000 7f6c06d4e640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.266+0000 7f6c06d4e640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.266+0000 7f6c06d4e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c000a8a30 0x7f6c000ac330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 10 monclient(hunting): picked mon.noname-c con 0x7f6c000a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c000a8e10 0x7f6c000ab920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 10 monclient(hunting): picked mon.noname-b con 0x7f6c000a8e10 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c001454a0 0x7f6c001da680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 10 monclient(hunting): picked mon.noname-a con 0x7f6c001454a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c000adaa0 con 0x7f6c001454a0 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c001da230 con 0x7f6c000a8e10 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:05.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c000b5720 con 0x7f6c000a8a30 2023-12-15T17:02:05.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:05.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.267+0000 7f6c06d4e640 10 monclient(hunting): authenticate will time out at 1429.584355s 2023-12-15T17:02:05.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.268+0000 7f8092d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:05.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.268+0000 7f8092d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:05.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.268+0000 7f8092d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:05.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.268+0000 7f8091d9b640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80940a8e10 0x7f809413b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.285 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.269+0000 7f9da77fe640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2023-12-15T17:02:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.269+0000 7f9da77fe640 1 -- 172.21.15.69:0/536006312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da81454a0 msgr2=0x7f9da8138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.269+0000 7f9da77fe640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da81454a0 0x7f9da8138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.269+0000 7f9da77fe640 1 -- 172.21.15.69:0/536006312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9da80a8a30 msgr2=0x7f9da80abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.269+0000 7f9da77fe640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9da80a8a30 0x7f9da80abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.269+0000 7f9da77fe640 10 monclient: _finish_hunting 0 2023-12-15T17:02:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.269+0000 7f9da77fe640 1 monclient: found mon.noname-c 2023-12-15T17:02:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.269+0000 7f9da77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.269+0000 7f9da77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.269+0000 7f9da77fe640 1 -- 172.21.15.69:0/536006312 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d980094d0 con 0x7f9da80a8e10 2023-12-15T17:02:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.271+0000 7f8091d9b640 10 monclient(hunting): get_auth_request con 0x7f80940a8e10 auth_method 0 2023-12-15T17:02:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.271+0000 7f8091d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.271+0000 7f8091d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.271+0000 7f8091d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.272+0000 7f6bfffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c000a8e10 0x7f6c000ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.272+0000 7f8091d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.272+0000 7f8091d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.272+0000 7f8091d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:05.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f6bfffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c000a8e10 0x7f6c000ab920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37488/0 (socket says 172.21.15.69:37488) 2023-12-15T17:02:05.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f6bfffff640 1 -- 172.21.15.69:0/865561889 learned_addr learned my addr 172.21.15.69:0/865561889 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:05.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f6bfffff640 10 monclient(hunting): get_auth_request con 0x7f6c000a8e10 auth_method 0 2023-12-15T17:02:05.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f6bfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:05.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f6bfffff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:05.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f6bfffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:05.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f8092d9d640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2023-12-15T17:02:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f8092d9d640 1 -- 172.21.15.69:0/1392961829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80940a8e10 msgr2=0x7f809413b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f8092d9d640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80940a8e10 0x7f809413b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f8092d9d640 1 -- 172.21.15.69:0/1392961829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80940a8a30 msgr2=0x7f80941415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f8092d9d640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80940a8a30 0x7f80941415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f8092d9d640 10 monclient: _finish_hunting 0 2023-12-15T17:02:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f8092d9d640 1 monclient: found mon.noname-c 2023-12-15T17:02:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f8092d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f8092d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.273+0000 7f8092d9d640 1 -- 172.21.15.69:0/1392961829 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8084009400 con 0x7f80941454a0 2023-12-15T17:02:05.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.275+0000 7f9da77fe640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8e10 0x7f9da80ac670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00b4d0 tx=0x7f9d9c00b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.275+0000 7f9da4ff9640 10 client.?.objecter ms_handle_connect 0x7f9da80a8e10 2023-12-15T17:02:05.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.275+0000 7f9da4ff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:05.290 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.275+0000 7f6c04aea640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c000a8a30 0x7f6c000ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.276+0000 7f8092d9d640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80941454a0 0x7f809413bbb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f808c00cab0 tx=0x7f808c00ceb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.277+0000 7f8082ffd640 10 client.?.objecter ms_handle_connect 0x7f80941454a0 2023-12-15T17:02:05.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.277+0000 7f8082ffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:05.291 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.278+0000 7f6c04aea640 10 monclient(hunting): get_auth_request con 0x7f6c000a8a30 auth_method 0 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.278+0000 7f6c04aea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.278+0000 7f6c04aea640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.278+0000 7f6c04aea640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.278+0000 7f6bfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.278+0000 7f6bfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.278+0000 7f6bfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.281+0000 7f6bfffff640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.281+0000 7f6bfffff640 1 -- 172.21.15.69:0/865561889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c000a8a30 msgr2=0x7f6c000ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.281+0000 7f6bfffff640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c000a8a30 0x7f6c000ac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.281+0000 7f6bfffff640 1 -- 172.21.15.69:0/865561889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c001454a0 msgr2=0x7f6c001da680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.281+0000 7f6bfffff640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c001454a0 0x7f6c001da680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.281+0000 7f6bfffff640 10 monclient: _finish_hunting 0 2023-12-15T17:02:05.292 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.281+0000 7f6bfffff640 1 monclient: found mon.noname-b 2023-12-15T17:02:05.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.281+0000 7f6bfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:05.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.281+0000 7f6bfffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:02:05.293 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.281+0000 7f6bfffff640 1 -- 172.21.15.69:0/865561889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf8009b30 con 0x7f6c000a8e10 2023-12-15T17:02:05.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9da4ff9640 1 -- 172.21.15.69:0/536006312 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c010870 con 0x7f9da80a8e10 2023-12-15T17:02:05.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9da4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9da4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:05.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9da4ff9640 10 monclient: dump: 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9da4ff9640 10 monclient: _finish_auth 0 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9da4ff9640 10 monclient: _check_auth_tickets 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9da4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:05.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9da4ff9640 1 -- 172.21.15.69:0/536006312 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9c0109d0 con 0x7f9da80a8e10 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9dae661640 5 monclient: authenticate success, global_id 4334 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9dae661640 10 monclient: _renew_subs 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9dae661640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9dae661640 1 -- 172.21.15.69:0/536006312 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da80acbb0 con 0x7f9da80a8e10 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9dae661640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9dae661640 10 monclient: _renew_subs 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9dae661640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9dae661640 1 -- 172.21.15.69:0/536006312 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da81393e0 con 0x7f9da80a8e10 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9dae661640 1 librados: init done 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9da4ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.288+0000 7f9da67fc640 4 set_mon_vals no callback set 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.289+0000 7f9da4ff9640 1 -- 172.21.15.69:0/536006312 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c010d80 con 0x7f9da80a8e10 2023-12-15T17:02:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.289+0000 7f9da4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.290+0000 7f9d8e7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.290+0000 7f9da4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.290+0000 7f9da4ff9640 10 monclient: dump: 2023-12-15T17:02:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.299 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.299 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.290+0000 7f9d8e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.290+0000 7f9d8e7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.290+0000 7f9d8e7fc640 1 -- 172.21.15.69:0/536006312 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d80005040 con 0x7f9da80a8e10 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 1 -- 172.21.15.69:0/1392961829 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c00ab70 con 0x7f80941454a0 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 10 monclient: dump: 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.300 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 1 -- 172.21.15.69:0/1392961829 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c00acd0 con 0x7f80941454a0 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 1 -- 172.21.15.69:0/1392961829 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c012600 con 0x7f80941454a0 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8082ffd640 10 monclient: dump: 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8098d48640 5 monclient: authenticate success, global_id 4337 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8098d48640 10 monclient: _renew_subs 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8083fff640 4 set_mon_vals no callback set 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8098d48640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8098d48640 1 -- 172.21.15.69:0/1392961829 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80941d6c10 con 0x7f80941454a0 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8098d48640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8098d48640 10 monclient: _renew_subs 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8098d48640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8098d48640 1 -- 172.21.15.69:0/1392961829 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809413c190 con 0x7f80941454a0 2023-12-15T17:02:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.291+0000 7f8098d48640 1 librados: init done 2023-12-15T17:02:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.292+0000 7f8098d48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.292+0000 7f8098d48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.292+0000 7f8098d48640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.292+0000 7f8098d48640 1 -- 172.21.15.69:0/1392961829 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8094067ac0 con 0x7f80941454a0 2023-12-15T17:02:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.296+0000 7f2afdffb640 1 -- 172.21.15.69:0/546692321 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2af8007470 con 0x7f2b080a8e10 2023-12-15T17:02:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.297+0000 7f2afdffb640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2aec06cba0 0x7f2aec06f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.297+0000 7f2b0d733640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2aec06cba0 0x7f2aec06f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.297+0000 7f2b0d733640 10 monclient: get_auth_request con 0x7f2aec06cba0 auth_method 0 2023-12-15T17:02:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.298+0000 7f2afdffb640 10 client.4353.objecter ms_handle_connect 0x7f2aec06cba0 2023-12-15T17:02:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.298+0000 7f2b0d733640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2aec06cba0 0x7f2aec06f060 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b040097a0 tx=0x7f2b04008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.299+0000 7f2afdffb640 1 -- 172.21.15.69:0/546692321 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2af8007720 con 0x7f2b080a8e10 2023-12-15T17:02:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.299+0000 7f2afdffb640 10 client.4353.objecter ms_dispatch 0x7f2b08000b90 osd_map(23..23 src has 1..23) v4 2023-12-15T17:02:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.299+0000 7f2afdffb640 3 client.4353.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:02:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.299+0000 7f2afdffb640 3 client.4353.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:02:05.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.300+0000 7f2afdffb640 20 client.4353.objecter dump_active .. 0 homeless 2023-12-15T17:02:05.332 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.330+0000 7f9da4ff9640 1 -- 172.21.15.69:0/536006312 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9d9c023480 con 0x7f9da80a8e10 2023-12-15T17:02:05.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.330+0000 7f6bfffff640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c000a8e10 0x7f6c000ab920 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00b4a0 tx=0x7f6bec00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.330+0000 7f6bfd7fa640 10 client.?.objecter ms_handle_connect 0x7f6c000a8e10 2023-12-15T17:02:05.333 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.331+0000 7f6bfd7fa640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:05.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.331+0000 7f9da4ff9640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9d9406cba0 0x7f9d9406f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.331+0000 7f9da7fff640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9d9406cba0 0x7f9d9406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.331+0000 7f9da7fff640 10 monclient: get_auth_request con 0x7f9d9406cba0 auth_method 0 2023-12-15T17:02:05.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.332+0000 7f9da7fff640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9d9406cba0 0x7f9d9406f060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9d98036000 tx=0x7f9d98030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.334 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.332+0000 7f9da4ff9640 10 client.4334.objecter ms_handle_connect 0x7f9d9406cba0 2023-12-15T17:02:05.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.341+0000 7fcbabfff640 1 -- 172.21.15.69:0/861125557 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcbb8021020 con 0x7fcbc40a8e10 2023-12-15T17:02:05.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.341+0000 7fcbabfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-15T17:02:05.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.341+0000 7fcbabfff640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:05.343 INFO:teuthology.orchestra.run.smithi069.stdout:64424509445 2023-12-15T17:02:05.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 10 librados: watch_flush enter 2023-12-15T17:02:05.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 10 librados: watch_flush exit 2023-12-15T17:02:05.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 20 client.4357.objecter shutdown clearing up homeless session... 2023-12-15T17:02:05.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 10 client.4357.objecter successfully canceled tick 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 -- 172.21.15.69:0/861125557 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcbb006cb50 msgr2=0x7fcbb006f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcbb006cb50 0x7fcbb006f010 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4036000 tx=0x7fcbb403c040 comp rx=0 tx=0).stop 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 10 monclient: shutdown 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 -- 172.21.15.69:0/861125557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbc40a8e10 msgr2=0x7fcbc40af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbc40a8e10 0x7fcbc40af6e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800e890 tx=0x7fcbb800ed50 comp rx=0 tx=0).stop 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 -- 172.21.15.69:0/861125557 shutdown_connections 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcbb006cb50 0x7fcbb006f010 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc41454a0 0x7fcbc41dac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbc40a8e10 0x7fcbc40af6e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 --2- 172.21.15.69:0/861125557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40a8a30 0x7fcbc40aec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 -- 172.21.15.69:0/861125557 >> 172.21.15.69:0/861125557 conn(0x7fcbc4061a30 msgr2=0x7fcbc413d060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:05.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 -- 172.21.15.69:0/861125557 shutdown_connections 2023-12-15T17:02:05.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 1 -- 172.21.15.69:0/865561889 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec010610 con 0x7f6c000a8e10 2023-12-15T17:02:05.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:05.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 10 monclient: dump: 2023-12-15T17:02:05.348 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.348 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.348 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.348 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.348 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.349 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.349 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.349 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.349 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.349 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 10 monclient: _finish_auth 0 2023-12-15T17:02:05.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 10 monclient: _check_auth_tickets 2023-12-15T17:02:05.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:05.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 -- 172.21.15.69:0/861125557 wait complete. 2023-12-15T17:02:05.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7fcbc9a39640 1 librados: shutdown 2023-12-15T17:02:05.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.345+0000 7f72af96c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-15T17:02:05.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.345+0000 7f72af96c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-15T17:02:05.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.345+0000 7f72af96c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:05.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.345+0000 7f72af96c640 1 -- 172.21.15.69:0/2577791817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f72a8075b60 con 0x7f72a8142a20 2023-12-15T17:02:05.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 1 -- 172.21.15.69:0/865561889 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bec010770 con 0x7f6c000a8e10 2023-12-15T17:02:05.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:05.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 1 -- 172.21.15.69:0/865561889 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec010b20 con 0x7f6c000a8e10 2023-12-15T17:02:05.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:05.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:05.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfd7fa640 10 monclient: dump: 2023-12-15T17:02:05.351 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:05.351 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:05.351 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.351 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6bfeffd640 4 set_mon_vals no callback set 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6c06d4e640 5 monclient: authenticate success, global_id 4372 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6c06d4e640 10 monclient: _renew_subs 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6c06d4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6c06d4e640 1 -- 172.21.15.69:0/865561889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c000adca0 con 0x7f6c000a8e10 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6c06d4e640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6c06d4e640 10 monclient: _renew_subs 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6c06d4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:05.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.344+0000 7f6c06d4e640 1 -- 172.21.15.69:0/865561889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c001dabc0 con 0x7f6c000a8e10 2023-12-15T17:02:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.345+0000 7f6c06d4e640 1 librados: init done 2023-12-15T17:02:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.345+0000 7f6c06d4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.345+0000 7f6c06d4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.345+0000 7f6c06d4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:05.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.345+0000 7f6c06d4e640 1 -- 172.21.15.69:0/865561889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c00067ac0 con 0x7f6c000a8e10 2023-12-15T17:02:05.358 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.2 2023-12-15T17:02:05.358 DEBUG:teuthology.parallel:result is None 2023-12-15T17:02:05.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.358+0000 7f83c077c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-15T17:02:05.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.358+0000 7f83c077c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-15T17:02:05.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.358+0000 7f83c077c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:05.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.358+0000 7f83c077c640 1 -- 172.21.15.69:0/1092473715 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f83b81ca0e0 con 0x7f83b80a8910 2023-12-15T17:02:05.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.479+0000 7f2afdffb640 1 -- 172.21.15.69:0/546692321 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2af8053ff0 con 0x7f2b080a8e10 2023-12-15T17:02:05.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.479+0000 7f2afdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.479+0000 7f2afdffb640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:05.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.487+0000 7f729dffb640 1 -- 172.21.15.69:0/2577791817 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f72a0019090 con 0x7f72a8142a20 2023-12-15T17:02:05.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.487+0000 7f729dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-15T17:02:05.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.487+0000 7f729dffb640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:05.489 INFO:teuthology.orchestra.run.smithi069.stdout:60129542149 2023-12-15T17:02:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 10 librados: watch_flush enter 2023-12-15T17:02:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 10 librados: watch_flush exit 2023-12-15T17:02:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 20 client.4344.objecter shutdown clearing up homeless session... 2023-12-15T17:02:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 10 client.4344.objecter successfully canceled tick 2023-12-15T17:02:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 -- 172.21.15.69:0/2577791817 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f728c06cba0 msgr2=0x7f728c06f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f728c06cba0 0x7f728c06f060 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72a4036000 tx=0x7f72a4008030 comp rx=0 tx=0).stop 2023-12-15T17:02:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 10 monclient: shutdown 2023-12-15T17:02:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 -- 172.21.15.69:0/2577791817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a8142a20 msgr2=0x7f72a8144d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a8142a20 0x7f72a8144d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f72a000b970 tx=0x7f72a000be30 comp rx=0 tx=0).stop 2023-12-15T17:02:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 -- 172.21.15.69:0/2577791817 shutdown_connections 2023-12-15T17:02:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f728c06cba0 0x7f728c06f060 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a8142a20 0x7f72a8144d80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72a80a8e10 0x7f72a8144840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 --2- 172.21.15.69:0/2577791817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72a80a8a30 0x7f72a81da600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 -- 172.21.15.69:0/2577791817 >> 172.21.15.69:0/2577791817 conn(0x7f72a8061a30 msgr2=0x7f72a80b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 -- 172.21.15.69:0/2577791817 shutdown_connections 2023-12-15T17:02:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 -- 172.21.15.69:0/2577791817 wait complete. 2023-12-15T17:02:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.490+0000 7f72af96c640 1 librados: shutdown 2023-12-15T17:02:05.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.498+0000 7f9ad2f2c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-15T17:02:05.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.498+0000 7f9ad2f2c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-15T17:02:05.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.498+0000 7f9ad2f2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:05.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.498+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/3086865092 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9acc075b60 con 0x7f9acc0a8a30 2023-12-15T17:02:05.506 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.7 2023-12-15T17:02:05.506 DEBUG:teuthology.parallel:result is None 2023-12-15T17:02:05.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.587+0000 7f6bfd7fa640 1 -- 172.21.15.69:0/865561889 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6bec010c80 con 0x7f6c000a8e10 2023-12-15T17:02:05.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.587+0000 7f6bfd7fa640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6be406cb50 0x7f6be406f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.587+0000 7f6c04aea640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6be406cb50 0x7f6be406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.587+0000 7f6c04aea640 10 monclient: get_auth_request con 0x7f6be406cb50 auth_method 0 2023-12-15T17:02:05.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.588+0000 7f6c04aea640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6be406cb50 0x7f6be406f010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009890 tx=0x7f6bf4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.588+0000 7f6bfd7fa640 10 client.4372.objecter ms_handle_connect 0x7f6be406cb50 2023-12-15T17:02:05.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.593+0000 7f6bfd7fa640 1 -- 172.21.15.69:0/865561889 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6bec058c40 con 0x7f6c000a8e10 2023-12-15T17:02:05.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.593+0000 7f6bfd7fa640 10 client.4372.objecter ms_dispatch 0x7f6c00000b90 osd_map(23..23 src has 1..23) v4 2023-12-15T17:02:05.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.593+0000 7f6bfd7fa640 3 client.4372.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:02:05.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.593+0000 7f6bfd7fa640 3 client.4372.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:02:05.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.593+0000 7f6bfd7fa640 20 client.4372.objecter dump_active .. 0 homeless 2023-12-15T17:02:05.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.663+0000 7f6bfd7fa640 1 -- 172.21.15.69:0/865561889 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6bec054880 con 0x7f6c000a8e10 2023-12-15T17:02:05.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.663+0000 7f6bfd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.663+0000 7f6bfd7fa640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:05.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.668+0000 7f2b0f997640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-15T17:02:05.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.668+0000 7f2b0f997640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-15T17:02:05.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.668+0000 7f2b0f997640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:05.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.668+0000 7f2b0f997640 1 -- 172.21.15.69:0/546692321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2b08142eb0 con 0x7f2b080a8e10 2023-12-15T17:02:05.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.672+0000 7f9ac97fa640 1 -- 172.21.15.69:0/3086865092 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9ac0019300 con 0x7f9acc0a8a30 2023-12-15T17:02:05.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.672+0000 7f9ac97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-15T17:02:05.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.672+0000 7f9ac97fa640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:05.674 INFO:teuthology.orchestra.run.smithi069.stdout:60129542151 2023-12-15T17:02:05.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 10 librados: watch_flush enter 2023-12-15T17:02:05.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 10 librados: watch_flush exit 2023-12-15T17:02:05.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 20 client.4366.objecter shutdown clearing up homeless session... 2023-12-15T17:02:05.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 10 client.4366.objecter successfully canceled tick 2023-12-15T17:02:05.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/3086865092 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ab006cb50 msgr2=0x7f9ab006f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ab006cb50 0x7f9ab006f010 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8036000 tx=0x7f9ab8039040 comp rx=0 tx=0).stop 2023-12-15T17:02:05.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 10 monclient: shutdown 2023-12-15T17:02:05.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:05.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/3086865092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9acc0a8a30 msgr2=0x7f9acc06da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9acc0a8a30 0x7f9acc06da90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9ac000c450 tx=0x7f9ac000c910 comp rx=0 tx=0).stop 2023-12-15T17:02:05.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/3086865092 shutdown_connections 2023-12-15T17:02:05.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ab006cb50 0x7f9ab006f010 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9acc142bf0 0x7f9acc1df150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9acc0a8e10 0x7f9acc1de960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 --2- 172.21.15.69:0/3086865092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9acc0a8a30 0x7f9acc06da90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/3086865092 >> 172.21.15.69:0/3086865092 conn(0x7f9acc061a30 msgr2=0x7f9acc13cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:05.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/3086865092 shutdown_connections 2023-12-15T17:02:05.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 -- 172.21.15.69:0/3086865092 wait complete. 2023-12-15T17:02:05.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.674+0000 7f9ad2f2c640 1 librados: shutdown 2023-12-15T17:02:05.686 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.5 2023-12-15T17:02:05.686 DEBUG:teuthology.parallel:result is None 2023-12-15T17:02:05.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.709+0000 7f2afdffb640 1 -- 172.21.15.69:0/546692321 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2b08142eb0 con 0x7f2b080a8e10 2023-12-15T17:02:05.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.709+0000 7f2afdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-15T17:02:05.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.709+0000 7f2afdffb640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:05.711 INFO:teuthology.orchestra.run.smithi069.stdout:60129542150 2023-12-15T17:02:05.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.710+0000 7f2b0f997640 10 librados: watch_flush enter 2023-12-15T17:02:05.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 10 librados: watch_flush exit 2023-12-15T17:02:05.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 20 client.4353.objecter shutdown clearing up homeless session... 2023-12-15T17:02:05.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 10 client.4353.objecter successfully canceled tick 2023-12-15T17:02:05.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 -- 172.21.15.69:0/546692321 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2aec06cba0 msgr2=0x7f2aec06f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2aec06cba0 0x7f2aec06f060 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b040097a0 tx=0x7f2b04008030 comp rx=0 tx=0).stop 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 10 monclient: shutdown 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 -- 172.21.15.69:0/546692321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b080a8e10 msgr2=0x7f2b081cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b080a8e10 0x7f2b081cece0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2af800c370 tx=0x7f2af800c830 comp rx=0 tx=0).stop 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 -- 172.21.15.69:0/546692321 shutdown_connections 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2aec06cba0 0x7f2aec06f060 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b08142a20 0x7f2b081db390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b080a8e10 0x7f2b081cece0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 --2- 172.21.15.69:0/546692321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b080a8a30 0x7f2b0813b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 -- 172.21.15.69:0/546692321 >> 172.21.15.69:0/546692321 conn(0x7f2b08061a30 msgr2=0x7f2b080b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 -- 172.21.15.69:0/546692321 shutdown_connections 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 -- 172.21.15.69:0/546692321 wait complete. 2023-12-15T17:02:05.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.711+0000 7f2b0f997640 1 librados: shutdown 2023-12-15T17:02:05.723 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.6 2023-12-15T17:02:05.723 DEBUG:teuthology.parallel:result is None 2023-12-15T17:02:05.865 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.863+0000 7f6c06d4e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-15T17:02:05.865 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.863+0000 7f6c06d4e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-15T17:02:05.865 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.863+0000 7f6c06d4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:05.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.863+0000 7f6c06d4e640 1 -- 172.21.15.69:0/865561889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6c00075b60 con 0x7f6c000a8e10 2023-12-15T17:02:05.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.867+0000 7f6bfd7fa640 1 -- 172.21.15.69:0/865561889 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6bec01e020 con 0x7f6c000a8e10 2023-12-15T17:02:05.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.867+0000 7f6bfd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-15T17:02:05.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.867+0000 7f6bfd7fa640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:05.869 INFO:teuthology.orchestra.run.smithi069.stdout:64424509447 2023-12-15T17:02:05.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.868+0000 7f6c06d4e640 10 librados: watch_flush enter 2023-12-15T17:02:05.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.868+0000 7f6c06d4e640 10 librados: watch_flush exit 2023-12-15T17:02:05.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.868+0000 7f6c06d4e640 20 client.4372.objecter shutdown clearing up homeless session... 2023-12-15T17:02:05.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.868+0000 7f6c06d4e640 10 client.4372.objecter successfully canceled tick 2023-12-15T17:02:05.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 -- 172.21.15.69:0/865561889 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6be406cb50 msgr2=0x7f6be406f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6be406cb50 0x7f6be406f010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009890 tx=0x7f6bf4008040 comp rx=0 tx=0).stop 2023-12-15T17:02:05.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 10 monclient: shutdown 2023-12-15T17:02:05.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:05.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 -- 172.21.15.69:0/865561889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c000a8e10 msgr2=0x7f6c000ab920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c000a8e10 0x7f6c000ab920 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00b4a0 tx=0x7f6bec00b960 comp rx=0 tx=0).stop 2023-12-15T17:02:05.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 -- 172.21.15.69:0/865561889 shutdown_connections 2023-12-15T17:02:05.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f6be406cb50 0x7f6be406f010 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c001454a0 0x7f6c001da680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c000a8e10 0x7f6c000ab920 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 --2- 172.21.15.69:0/865561889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c000a8a30 0x7f6c000ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 -- 172.21.15.69:0/865561889 >> 172.21.15.69:0/865561889 conn(0x7f6c00061a30 msgr2=0x7f6c0013ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:05.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 -- 172.21.15.69:0/865561889 shutdown_connections 2023-12-15T17:02:05.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 -- 172.21.15.69:0/865561889 wait complete. 2023-12-15T17:02:05.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.869+0000 7f6c06d4e640 1 librados: shutdown 2023-12-15T17:02:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.877+0000 7f8082ffd640 1 -- 172.21.15.69:0/1392961829 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f808c012760 con 0x7f80941454a0 2023-12-15T17:02:05.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.877+0000 7f8082ffd640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f807406cb50 0x7f807406f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:05.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.877+0000 7f809259c640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f807406cb50 0x7f807406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:05.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.877+0000 7f809259c640 10 monclient: get_auth_request con 0x7f807406cb50 auth_method 0 2023-12-15T17:02:05.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.878+0000 7f809259c640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f807406cb50 0x7f807406f010 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8084036000 tx=0x7f8084030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.878+0000 7f8082ffd640 10 client.4337.objecter ms_handle_connect 0x7f807406cb50 2023-12-15T17:02:05.883 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.1 2023-12-15T17:02:05.883 DEBUG:teuthology.parallel:result is None 2023-12-15T17:02:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.898+0000 7f9da4ff9640 1 -- 172.21.15.69:0/536006312 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9d9c058ed0 con 0x7f9da80a8e10 2023-12-15T17:02:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.898+0000 7f9da4ff9640 10 client.4334.objecter ms_dispatch 0x7f9da8000b90 osd_map(23..23 src has 1..23) v4 2023-12-15T17:02:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.898+0000 7f9da4ff9640 3 client.4334.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:02:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.898+0000 7f9da4ff9640 3 client.4334.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:02:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.898+0000 7f9da4ff9640 20 client.4334.objecter dump_active .. 0 homeless 2023-12-15T17:02:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.898+0000 7f9da4ff9640 1 -- 172.21.15.69:0/536006312 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9d9c054b10 con 0x7f9da80a8e10 2023-12-15T17:02:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.898+0000 7f9da4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.898+0000 7f9da4ff9640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:05.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.909+0000 7f8082ffd640 1 -- 172.21.15.69:0/1392961829 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f808c00ae40 con 0x7f80941454a0 2023-12-15T17:02:05.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.909+0000 7f8082ffd640 10 client.4337.objecter ms_dispatch 0x7f8094000b90 osd_map(23..23 src has 1..23) v4 2023-12-15T17:02:05.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.909+0000 7f8082ffd640 3 client.4337.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:02:05.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.909+0000 7f8082ffd640 3 client.4337.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:02:05.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.909+0000 7f8082ffd640 20 client.4337.objecter dump_active .. 0 homeless 2023-12-15T17:02:05.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.909+0000 7f8082ffd640 1 -- 172.21.15.69:0/1392961829 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f808c054d70 con 0x7f80941454a0 2023-12-15T17:02:05.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.909+0000 7f8082ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:05.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.909+0000 7f8082ffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:05.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.916+0000 7f83aeffd640 1 -- 172.21.15.69:0/1092473715 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f83b81ca0e0 con 0x7f83b80a8910 2023-12-15T17:02:05.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.916+0000 7f83aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-15T17:02:05.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.916+0000 7f83aeffd640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:05.918 INFO:teuthology.orchestra.run.smithi069.stdout:64424509447 2023-12-15T17:02:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.918+0000 7f83c077c640 10 librados: watch_flush enter 2023-12-15T17:02:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.918+0000 7f83c077c640 10 librados: watch_flush exit 2023-12-15T17:02:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.918+0000 7f83c077c640 20 client.4319.objecter shutdown clearing up homeless session... 2023-12-15T17:02:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.918+0000 7f83c077c640 10 client.4319.objecter successfully canceled tick 2023-12-15T17:02:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 -- 172.21.15.69:0/1092473715 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f839c06cba0 msgr2=0x7f839c06f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f839c06cba0 0x7f839c06f060 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f83b4036000 tx=0x7f83b4030000 comp rx=0 tx=0).stop 2023-12-15T17:02:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 10 monclient: shutdown 2023-12-15T17:02:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 -- 172.21.15.69:0/1092473715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a8910 msgr2=0x7f83b81c86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a8910 0x7f83b81c86e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f83a800b410 tx=0x7f83a800b8d0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 -- 172.21.15.69:0/1092473715 shutdown_connections 2023-12-15T17:02:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f839c06cba0 0x7f839c06f060 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b813b0d0 0x7f83b81d72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83b8141f90 0x7f83b813ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 --2- 172.21.15.69:0/1092473715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b80a8910 0x7f83b81c86e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 -- 172.21.15.69:0/1092473715 >> 172.21.15.69:0/1092473715 conn(0x7f83b80b7780 msgr2=0x7f83b813c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 -- 172.21.15.69:0/1092473715 shutdown_connections 2023-12-15T17:02:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 -- 172.21.15.69:0/1092473715 wait complete. 2023-12-15T17:02:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:05.919+0000 7f83c077c640 1 librados: shutdown 2023-12-15T17:02:05.935 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.0 2023-12-15T17:02:05.936 DEBUG:teuthology.parallel:result is None 2023-12-15T17:02:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.071+0000 7f9d8e7fc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-15T17:02:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.071+0000 7f9d8e7fc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-15T17:02:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.071+0000 7f9d8e7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.071+0000 7f9d8e7fc640 1 -- 172.21.15.69:0/536006312 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9d800055c0 con 0x7f9da80a8e10 2023-12-15T17:02:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.076+0000 7f9da4ff9640 1 -- 172.21.15.69:0/536006312 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9d9c01e020 con 0x7f9da80a8e10 2023-12-15T17:02:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.076+0000 7f9da4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-15T17:02:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.076+0000 7f9da4ff9640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:06.078 INFO:teuthology.orchestra.run.smithi069.stdout:60129542151 2023-12-15T17:02:06.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 10 librados: watch_flush enter 2023-12-15T17:02:06.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 10 librados: watch_flush exit 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 20 client.4334.objecter shutdown clearing up homeless session... 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 10 client.4334.objecter successfully canceled tick 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 -- 172.21.15.69:0/536006312 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9d9406cba0 msgr2=0x7f9d9406f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9d9406cba0 0x7f9d9406f060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9d98036000 tx=0x7f9d98030000 comp rx=0 tx=0).stop 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 10 monclient: shutdown 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 -- 172.21.15.69:0/536006312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8e10 msgr2=0x7f9da80ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8e10 0x7f9da80ac670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00b4d0 tx=0x7f9d9c00b990 comp rx=0 tx=0).stop 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 -- 172.21.15.69:0/536006312 shutdown_connections 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9d9406cba0 0x7f9d9406f060 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da81454a0 0x7f9da8138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9da80a8e10 0x7f9da80ac670 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 --2- 172.21.15.69:0/536006312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9da80a8a30 0x7f9da80abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 -- 172.21.15.69:0/536006312 >> 172.21.15.69:0/536006312 conn(0x7f9da8061a30 msgr2=0x7f9da813cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 -- 172.21.15.69:0/536006312 shutdown_connections 2023-12-15T17:02:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 -- 172.21.15.69:0/536006312 wait complete. 2023-12-15T17:02:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.078+0000 7f9d8e7fc640 1 librados: shutdown 2023-12-15T17:02:06.095 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.4 2023-12-15T17:02:06.095 DEBUG:teuthology.parallel:result is None 2023-12-15T17:02:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.099+0000 7f8098d48640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-15T17:02:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.099+0000 7f8098d48640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-15T17:02:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.099+0000 7f8098d48640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:02:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.099+0000 7f8098d48640 1 -- 172.21.15.69:0/1392961829 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f809413c3d0 con 0x7f80941454a0 2023-12-15T17:02:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.102+0000 7f8082ffd640 1 -- 172.21.15.69:0/1392961829 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f809413c3d0 con 0x7f80941454a0 2023-12-15T17:02:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.103+0000 7f8082ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-15T17:02:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.103+0000 7f8082ffd640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:06.105 INFO:teuthology.orchestra.run.smithi069.stdout:64424509445 2023-12-15T17:02:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.104+0000 7f8098d48640 10 librados: watch_flush enter 2023-12-15T17:02:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 10 librados: watch_flush exit 2023-12-15T17:02:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 20 client.4337.objecter shutdown clearing up homeless session... 2023-12-15T17:02:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 10 client.4337.objecter successfully canceled tick 2023-12-15T17:02:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 -- 172.21.15.69:0/1392961829 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f807406cb50 msgr2=0x7f807406f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f807406cb50 0x7f807406f010 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8084036000 tx=0x7f8084030000 comp rx=0 tx=0).stop 2023-12-15T17:02:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 10 monclient: shutdown 2023-12-15T17:02:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 -- 172.21.15.69:0/1392961829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80941454a0 msgr2=0x7f809413bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80941454a0 0x7f809413bbb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f808c00cab0 tx=0x7f808c00ceb0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 -- 172.21.15.69:0/1392961829 shutdown_connections 2023-12-15T17:02:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f807406cb50 0x7f807406f010 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80941454a0 0x7f809413bbb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80940a8e10 0x7f809413b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 --2- 172.21.15.69:0/1392961829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80940a8a30 0x7f80941415e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 -- 172.21.15.69:0/1392961829 >> 172.21.15.69:0/1392961829 conn(0x7f8094061a30 msgr2=0x7f809413cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 -- 172.21.15.69:0/1392961829 shutdown_connections 2023-12-15T17:02:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 -- 172.21.15.69:0/1392961829 wait complete. 2023-12-15T17:02:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.105+0000 7f8098d48640 1 librados: shutdown 2023-12-15T17:02:06.118 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.3 2023-12-15T17:02:06.118 DEBUG:teuthology.parallel:result is None 2023-12-15T17:02:06.118 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-15T17:02:06.118 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T17:02:06.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.257+0000 7f08bc378640 10 monclient: get_monmap_and_config 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.257+0000 7f08bc378640 10 monclient: build_initial_monmap 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.257+0000 7f08bc378640 10 monclient: monmap: 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:06.258532+0000 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:06.258532+0000 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:06.259 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.257+0000 7f08bc378640 1 Processor -- start 2023-12-15T17:02:06.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.257+0000 7f08bc378640 1 -- start start 2023-12-15T17:02:06.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.257+0000 7f08bc378640 10 monclient: init 2023-12-15T17:02:06.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:06.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-12-15T17:02:06.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08b40a8a30 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:06.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f08b4142bf0 0x7f08b413ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:06.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 10 monclient(hunting): picked mon.noname-a con 0x7f08b4142bf0 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 0x7f08b413f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 10 monclient(hunting): picked mon.noname-b con 0x7f08b40a8e10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b40aac10 con 0x7f08b40a8a30 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b40aaa10 con 0x7f08b4142bf0 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40aaef0 con 0x7f08b40a8e10 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.258+0000 7f08bc378640 10 monclient(hunting): authenticate will time out at 1430.574863s 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08b9913640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f08b4142bf0 0x7f08b413ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08ba915640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 0x7f08b413f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08b9913640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f08b4142bf0 0x7f08b413ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58836/0 (socket says 172.21.15.69:58836) 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08b9913640 1 -- 172.21.15.69:0/1419337597 learned_addr learned my addr 172.21.15.69:0/1419337597 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08b9913640 10 monclient(hunting): get_auth_request con 0x7f08b4142bf0 auth_method 0 2023-12-15T17:02:06.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08b9913640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08b9913640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08b9913640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08ba915640 10 monclient(hunting): get_auth_request con 0x7f08b40a8e10 auth_method 0 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08ba915640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08ba915640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08ba915640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08b9913640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08b9913640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08b9913640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08ba915640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.259+0000 7f08ba915640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:06.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.260+0000 7f08ba915640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:06.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.261+0000 7f08ba915640 10 monclient(hunting): handle_auth_done global_id 4343 payload 274 2023-12-15T17:02:06.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.261+0000 7f08ba915640 1 -- 172.21.15.69:0/1419337597 >> v2:172.21.15.69:3300/0 conn(0x7f08b4142bf0 msgr2=0x7f08b413ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:06.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.261+0000 7f08ba915640 1 --2- 172.21.15.69:0/1419337597 >> v2:172.21.15.69:3300/0 conn(0x7f08b4142bf0 0x7f08b413ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.261+0000 7f08ba915640 1 -- 172.21.15.69:0/1419337597 >> v1:172.21.15.69:6789/0 conn(0x7f08b40a8a30 legacy=0x7f08b41424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:06.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.261+0000 7f08ba915640 10 monclient: _finish_hunting 0 2023-12-15T17:02:06.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.261+0000 7f08ba915640 1 monclient: found mon.noname-b 2023-12-15T17:02:06.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.261+0000 7f08ba915640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:06.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.261+0000 7f08ba915640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:02:06.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.261+0000 7f08ba915640 1 -- 172.21.15.69:0/1419337597 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b40a97f0 con 0x7f08b40a8e10 2023-12-15T17:02:06.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.263+0000 7f08ba915640 1 --2- 172.21.15.69:0/1419337597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 0x7f08b413f230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009d90 tx=0x7f08ac02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21a09b916808d0b4 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:06.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.266+0000 7f08b9112640 1 -- 172.21.15.69:0/1419337597 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac02cbc0 con 0x7f08b40a8e10 2023-12-15T17:02:06.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:06.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:06.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 10 monclient: dump: 2023-12-15T17:02:06.269 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 10 monclient: _finish_auth 0 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 10 monclient: _check_auth_tickets 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 1 -- 172.21.15.69:0/1419337597 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac02cd20 con 0x7f08b40a8e10 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 5 monclient: authenticate success, global_id 4343 2023-12-15T17:02:06.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 1 -- 172.21.15.69:0/1419337597 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac034660 con 0x7f08b40a8e10 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 4 set_mon_vals no callback set 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08b9112640 10 monclient: dump: 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:06.272 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 10 monclient: shutdown 2023-12-15T17:02:06.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:06.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 1 -- 172.21.15.69:0/1419337597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 msgr2=0x7f08b413f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 1 --2- 172.21.15.69:0/1419337597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 0x7f08b413f230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009d90 tx=0x7f08ac02be10 comp rx=0 tx=0).stop 2023-12-15T17:02:06.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 1 -- 172.21.15.69:0/1419337597 shutdown_connections 2023-12-15T17:02:06.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 1 --2- 172.21.15.69:0/1419337597 >> v2:172.21.15.69:3300/0 conn(0x7f08b4142bf0 0x7f08b413ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 1 --2- 172.21.15.69:0/1419337597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 0x7f08b413f230 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 1 -- 172.21.15.69:0/1419337597 >> 172.21.15.69:0/1419337597 conn(0x7f08b4061a30 msgr2=0x7f08b40b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:06.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 1 -- 172.21.15.69:0/1419337597 shutdown_connections 2023-12-15T17:02:06.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.267+0000 7f08bc378640 1 -- 172.21.15.69:0/1419337597 wait complete. 2023-12-15T17:02:06.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.268+0000 7f08bc378640 10 monclient: build_initial_monmap 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.268+0000 7f08bc378640 10 monclient: monmap: 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 librados: starting msgr at 2023-12-15T17:02:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 librados: starting objecter 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 Processor -- start 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 -- start start 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 librados: setting wanted keys 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 librados: calling monclient init 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 10 monclient: init 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08b40a8a30 0x7f08b40ae940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 10 monclient(hunting): picked mon.noname-a con 0x7f08b40a8a30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 0x7f08b40adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 10 monclient(hunting): picked mon.noname-c con 0x7f08b40a8e10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b4142bf0 0x7f08b41da840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 10 monclient(hunting): picked mon.noname-b con 0x7f08b4142bf0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40af060 con 0x7f08b40a8a30 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40b0520 con 0x7f08b4142bf0 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40acdf0 con 0x7f08b40a8e10 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.269+0000 7f08bc378640 10 monclient(hunting): authenticate will time out at 1430.586265s 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08b9913640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 0x7f08b40adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08ba114640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08b40a8a30 0x7f08b40ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08ba915640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b4142bf0 0x7f08b41da840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08b9913640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 0x7f08b40adc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53148/0 (socket says 172.21.15.69:53148) 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08b9913640 1 -- 172.21.15.69:0/4237094077 learned_addr learned my addr 172.21.15.69:0/4237094077 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08b9913640 10 monclient(hunting): get_auth_request con 0x7f08b40a8e10 auth_method 0 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08b9913640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08b9913640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08b9913640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08ba114640 10 monclient(hunting): get_auth_request con 0x7f08b40a8a30 auth_method 0 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08ba114640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08ba114640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08ba114640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08ba915640 10 monclient(hunting): get_auth_request con 0x7f08b4142bf0 auth_method 0 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08ba915640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08ba915640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.270+0000 7f08ba915640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.271+0000 7f08b9913640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.271+0000 7f08b9913640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.271+0000 7f08b9913640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.271+0000 7f08ba114640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.271+0000 7f08ba114640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.271+0000 7f08ba114640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.271+0000 7f08ba915640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.271+0000 7f08ba915640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.271+0000 7f08ba915640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.273+0000 7f08ba915640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2023-12-15T17:02:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.273+0000 7f08ba915640 1 -- 172.21.15.69:0/4237094077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 msgr2=0x7f08b40adc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.273+0000 7f08ba915640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 0x7f08b40adc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.273+0000 7f08ba915640 1 -- 172.21.15.69:0/4237094077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08b40a8a30 msgr2=0x7f08b40ae940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.273+0000 7f08ba915640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08b40a8a30 0x7f08b40ae940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.273+0000 7f08ba915640 10 monclient: _finish_hunting 0 2023-12-15T17:02:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.273+0000 7f08ba915640 1 monclient: found mon.noname-b 2023-12-15T17:02:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.273+0000 7f08ba915640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:06.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.273+0000 7f08ba915640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:02:06.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.273+0000 7f08ba915640 1 -- 172.21.15.69:0/4237094077 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac009a70 con 0x7f08b4142bf0 2023-12-15T17:02:06.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.276+0000 7f08ba915640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b4142bf0 0x7f08b41da840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f08ac033000 tx=0x7f08ac004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:06.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.276+0000 7f08aaffd640 10 client.?.objecter ms_handle_connect 0x7f08b4142bf0 2023-12-15T17:02:06.283 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.276+0000 7f08aaffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:06.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 1 -- 172.21.15.69:0/4237094077 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac03e610 con 0x7f08b4142bf0 2023-12-15T17:02:06.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:06.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:06.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 10 monclient: dump: 2023-12-15T17:02:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:06.286 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 1 -- 172.21.15.69:0/4237094077 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac02ce90 con 0x7f08b4142bf0 2023-12-15T17:02:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 1 -- 172.21.15.69:0/4237094077 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac03e9d0 con 0x7f08b4142bf0 2023-12-15T17:02:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08aaffd640 10 monclient: dump: 2023-12-15T17:02:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08bc378640 5 monclient: authenticate success, global_id 4375 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08bc378640 10 monclient: _renew_subs 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08abfff640 4 set_mon_vals no callback set 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08bc378640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08bc378640 1 -- 172.21.15.69:0/4237094077 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b40ae170 con 0x7f08b4142bf0 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08bc378640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08bc378640 10 monclient: _renew_subs 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08bc378640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08bc378640 1 -- 172.21.15.69:0/4237094077 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b41daf60 con 0x7f08b4142bf0 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.279+0000 7f08bc378640 1 librados: init done 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.280+0000 7f08bc378640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.280+0000 7f08bc378640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.280+0000 7f08bc378640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.280+0000 7f08bc378640 1 -- 172.21.15.69:0/4237094077 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b4067ac0 con 0x7f08b4142bf0 2023-12-15T17:02:06.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.302+0000 7f08aaffd640 1 -- 172.21.15.69:0/4237094077 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f08ac03b050 con 0x7f08b4142bf0 2023-12-15T17:02:06.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.302+0000 7f08aaffd640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f089806cba0 0x7f089806f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:06.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.302+0000 7f08ba114640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f089806cba0 0x7f089806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:06.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.302+0000 7f08ba114640 10 monclient: get_auth_request con 0x7f089806cba0 auth_method 0 2023-12-15T17:02:06.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.303+0000 7f08aaffd640 10 client.4375.objecter ms_handle_connect 0x7f089806cba0 2023-12-15T17:02:06.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.303+0000 7f08ba114640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f089806cba0 0x7f089806f060 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f08b0004f60 tx=0x7f08b0007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:06.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.304+0000 7f08aaffd640 1 -- 172.21.15.69:0/4237094077 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f08ac07af70 con 0x7f08b4142bf0 2023-12-15T17:02:06.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.304+0000 7f08aaffd640 10 client.4375.objecter ms_dispatch 0x7f08b4000b90 osd_map(23..23 src has 1..23) v4 2023-12-15T17:02:06.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.304+0000 7f08aaffd640 3 client.4375.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:02:06.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.304+0000 7f08aaffd640 3 client.4375.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:02:06.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.305+0000 7f08aaffd640 20 client.4375.objecter dump_active .. 0 homeless 2023-12-15T17:02:06.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.541+0000 7f08aaffd640 1 -- 172.21.15.69:0/4237094077 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f08ac0772b0 con 0x7f08b4142bf0 2023-12-15T17:02:06.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.541+0000 7f08aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:06.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.541+0000 7f08aaffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:06.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.744+0000 7f08bc378640 1 -- 172.21.15.69:0/4237094077 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08b413f770 con 0x7f089806cba0 2023-12-15T17:02:06.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.747+0000 7f08aaffd640 1 -- 172.21.15.69:0/4237094077 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48069 (secure 0 0 0) 0x7f08b0003940 con 0x7f089806cba0 2023-12-15T17:02:06.749 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:02:06.749 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-12-15T17:02:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 10 librados: watch_flush enter 2023-12-15T17:02:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 10 librados: watch_flush exit 2023-12-15T17:02:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 20 client.4375.objecter shutdown clearing up homeless session... 2023-12-15T17:02:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 10 client.4375.objecter successfully canceled tick 2023-12-15T17:02:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 -- 172.21.15.69:0/4237094077 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f089806cba0 msgr2=0x7f089806f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f089806cba0 0x7f089806f060 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f08b0004f60 tx=0x7f08b0007e30 comp rx=0 tx=0).stop 2023-12-15T17:02:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 10 monclient: shutdown 2023-12-15T17:02:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 -- 172.21.15.69:0/4237094077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b4142bf0 msgr2=0x7f08b41da840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b4142bf0 0x7f08b41da840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f08ac033000 tx=0x7f08ac004a40 comp rx=0 tx=0).stop 2023-12-15T17:02:06.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 -- 172.21.15.69:0/4237094077 shutdown_connections 2023-12-15T17:02:06.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f089806cba0 0x7f089806f060 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b4142bf0 0x7f08b41da840 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08b40a8e10 0x7f08b40adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 --2- 172.21.15.69:0/4237094077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08b40a8a30 0x7f08b40ae940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 -- 172.21.15.69:0/4237094077 >> 172.21.15.69:0/4237094077 conn(0x7f08b4061a30 msgr2=0x7f08b413cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:06.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 -- 172.21.15.69:0/4237094077 shutdown_connections 2023-12-15T17:02:06.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 -- 172.21.15.69:0/4237094077 wait complete. 2023-12-15T17:02:06.752 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.748+0000 7f08bc378640 1 librados: shutdown 2023-12-15T17:02:06.761 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-12-15T17:02:05.438770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":111,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218876,"kb_used_data":2524,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755844,"statfs":{"total":773094113280,"available":772869984256,"internally_reserved":0,"allocated":2584576,"data_stored":1401002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":119,"apply_latency_ms":119,"commit_latency_ns":119000000,"apply_latency_ns":119000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":12,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000947"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-15T17:02:01.308854+0000","last_change":"2023-12-15T17:02:01.284924+0000","last_active":"2023-12-15T17:02:01.308854+0000","last_peered":"2023-12-15T17:02:01.308854+0000","last_clean":"2023-12-15T17:02:01.308854+0000","last_became_active":"2023-12-15T17:02:01.275938+0000","last_became_peered":"2023-12-15T17:02:01.275938+0000","last_unstale":"2023-12-15T17:02:01.308854+0000","last_undegraded":"2023-12-15T17:02:01.308854+0000","last_fullsized":"2023-12-15T17:02:01.308854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:35:22.642595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-15T17:02:03.938406+0000","last_change":"2023-12-15T17:01:59.949551+0000","last_active":"2023-12-15T17:02:03.938406+0000","last_peered":"2023-12-15T17:02:03.938406+0000","last_clean":"2023-12-15T17:02:03.938406+0000","last_became_active":"2023-12-15T17:01:59.918026+0000","last_became_peered":"2023-12-15T17:01:59.918026+0000","last_unstale":"2023-12-15T17:02:03.938406+0000","last_undegraded":"2023-12-15T17:02:03.938406+0000","last_fullsized":"2023-12-15T17:02:03.938406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:10:51.112278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-15T17:02:01.408317+0000","last_change":"2023-12-15T17:02:01.380902+0000","last_active":"2023-12-15T17:02:01.408317+0000","last_peered":"2023-12-15T17:02:01.408317+0000","last_clean":"2023-12-15T17:02:01.408317+0000","last_became_active":"2023-12-15T17:02:01.375672+0000","last_became_peered":"2023-12-15T17:02:01.375672+0000","last_unstale":"2023-12-15T17:02:01.408317+0000","last_undegraded":"2023-12-15T17:02:01.408317+0000","last_fullsized":"2023-12-15T17:02:01.408317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:15:10.893330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-15T17:02:04.067403+0000","last_change":"2023-12-15T17:01:59.938931+0000","last_active":"2023-12-15T17:02:04.067403+0000","last_peered":"2023-12-15T17:02:04.067403+0000","last_clean":"2023-12-15T17:02:04.067403+0000","last_became_active":"2023-12-15T17:01:59.908680+0000","last_became_peered":"2023-12-15T17:01:59.908680+0000","last_unstale":"2023-12-15T17:02:04.067403+0000","last_undegraded":"2023-12-15T17:02:04.067403+0000","last_fullsized":"2023-12-15T17:02:04.067403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:35:37.366491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-15T17:02:01.718405+0000","last_change":"2023-12-15T17:02:01.718405+0000","last_active":"2023-12-15T17:02:01.718405+0000","last_peered":"2023-12-15T17:02:01.718405+0000","last_clean":"2023-12-15T17:02:01.718405+0000","last_became_active":"2023-12-15T17:02:01.713188+0000","last_became_peered":"2023-12-15T17:02:01.713188+0000","last_unstale":"2023-12-15T17:02:01.718405+0000","last_undegraded":"2023-12-15T17:02:01.718405+0000","last_fullsized":"2023-12-15T17:02:01.718405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:56:55.902253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-15T17:02:03.965576+0000","last_change":"2023-12-15T17:01:59.559261+0000","last_active":"2023-12-15T17:02:03.965576+0000","last_peered":"2023-12-15T17:02:03.965576+0000","last_clean":"2023-12-15T17:02:03.965576+0000","last_became_active":"2023-12-15T17:01:59.531480+0000","last_became_peered":"2023-12-15T17:01:59.531480+0000","last_unstale":"2023-12-15T17:02:03.965576+0000","last_undegraded":"2023-12-15T17:02:03.965576+0000","last_fullsized":"2023-12-15T17:02:03.965576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:51:07.527384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-15T17:02:01.306086+0000","last_change":"2023-12-15T17:02:01.240306+0000","last_active":"2023-12-15T17:02:01.306086+0000","last_peered":"2023-12-15T17:02:01.306086+0000","last_clean":"2023-12-15T17:02:01.306086+0000","last_became_active":"2023-12-15T17:02:01.208353+0000","last_became_peered":"2023-12-15T17:02:01.208353+0000","last_unstale":"2023-12-15T17:02:01.306086+0000","last_undegraded":"2023-12-15T17:02:01.306086+0000","last_fullsized":"2023-12-15T17:02:01.306086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T00:32:59.159368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-15T17:02:01.625426+0000","last_change":"2023-12-15T17:02:01.625426+0000","last_active":"2023-12-15T17:02:01.625426+0000","last_peered":"2023-12-15T17:02:01.625426+0000","last_clean":"2023-12-15T17:02:01.625426+0000","last_became_active":"2023-12-15T17:02:01.616517+0000","last_became_peered":"2023-12-15T17:02:01.616517+0000","last_unstale":"2023-12-15T17:02:01.625426+0000","last_undegraded":"2023-12-15T17:02:01.625426+0000","last_fullsized":"2023-12-15T17:02:01.625426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:10:18.201816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'75","reported_seq":118,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-15T17:02:02.580906+0000","last_change":"2023-12-15T17:01:52.907228+0000","last_active":"2023-12-15T17:02:02.580906+0000","last_peered":"2023-12-15T17:02:02.580906+0000","last_clean":"2023-12-15T17:02:02.580906+0000","last_became_active":"2023-12-15T17:01:52.878062+0000","last_became_peered":"2023-12-15T17:01:52.878062+0000","last_unstale":"2023-12-15T17:02:02.580906+0000","last_undegraded":"2023-12-15T17:02:02.580906+0000","last_fullsized":"2023-12-15T17:02:02.580906+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_clean_scrub_stamp":"2023-12-15T17:01:50.252516+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:47:14.717362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":111,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":111,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":74820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0850000000000009}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1180000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.824999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.739999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.972000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.884}]}]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":610304,"data_stored":467250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.6600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.105}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.127000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.715999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.614000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.98}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.339}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.076000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.282}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":52.637999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7449999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.155999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":35.234000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.366}]}]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":69410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.554}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.507000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.728000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.441000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.773}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.808}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.048}]}]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":80568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6589999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.327000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8629999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6890000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.048999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.289000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.186}]}]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":80568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.801}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.823}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.814}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.448}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.641999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.673999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.606}]}]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":634880,"data_stored":478408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9619999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9580000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5519999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.282}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.952000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.558999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":80568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1429999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9749999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.827}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.817}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.305}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.532}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-15T17:02:06.764 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T17:02:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.861+0000 7fe776d25640 10 monclient: get_monmap_and_config 2023-12-15T17:02:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.861+0000 7fe776d25640 10 monclient: build_initial_monmap 2023-12-15T17:02:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.861+0000 7fe776d25640 10 monclient: monmap: 2023-12-15T17:02:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:06.862812+0000 2023-12-15T17:02:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:06.862812+0000 2023-12-15T17:02:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:06.864 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 1 Processor -- start 2023-12-15T17:02:06.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 1 -- start start 2023-12-15T17:02:06.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 10 monclient: init 2023-12-15T17:02:06.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:06.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-12-15T17:02:06.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe770060f20 0x7fe7700ad660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:06.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 10 monclient(hunting): picked mon.noname-a con 0x7fe770060f20 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:06.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe770061300 0x7fe7701430f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:06.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 10 monclient(hunting): picked mon.noname-b con 0x7fe770061300 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:06.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7700adba0 addr v1:172.21.15.192:6789/0 2023-12-15T17:02:06.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7700a8660 con 0x7fe7700adba0 2023-12-15T17:02:06.867 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7700af300 con 0x7fe770060f20 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7700a8940 con 0x7fe770061300 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.862+0000 7fe776d25640 10 monclient(hunting): authenticate will time out at 1431.179136s 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe76ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe770061300 0x7fe7701430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe774ac1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe770060f20 0x7fe7700ad660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe7752c2640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fe7700adba0 0x7fe770143630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:60626/0 (socket says 172.21.15.69:60626) 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe7752c2640 1 -- 172.21.15.69:0/1968139617 learned_addr learned my addr 172.21.15.69:0/1968139617 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe76ffff640 10 monclient(hunting): get_auth_request con 0x7fe770061300 auth_method 0 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe76ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe76ffff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe76ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:06.868 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe774ac1640 10 monclient(hunting): get_auth_request con 0x7fe770060f20 auth_method 0 2023-12-15T17:02:06.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe774ac1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:06.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe774ac1640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:06.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.863+0000 7fe774ac1640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:06.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.864+0000 7fe76ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:06.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.864+0000 7fe76ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:06.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.864+0000 7fe76ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:06.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.864+0000 7fe774ac1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:06.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.864+0000 7fe774ac1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:06.869 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.864+0000 7fe774ac1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:06.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.865+0000 7fe76f7fe640 1 -- 172.21.15.69:0/1968139617 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886505466 0 0) 0x7fe7700a8660 con 0x7fe7700adba0 2023-12-15T17:02:06.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.865+0000 7fe774ac1640 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2023-12-15T17:02:06.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.865+0000 7fe774ac1640 1 -- 172.21.15.69:0/1968139617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe770061300 msgr2=0x7fe7701430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:06.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.865+0000 7fe774ac1640 1 --2- 172.21.15.69:0/1968139617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe770061300 0x7fe7701430f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.865+0000 7fe774ac1640 1 -- 172.21.15.69:0/1968139617 >> v1:172.21.15.192:6789/0 conn(0x7fe7700adba0 legacy=0x7fe770143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.865+0000 7fe774ac1640 10 monclient: _finish_hunting 0 2023-12-15T17:02:06.870 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.865+0000 7fe774ac1640 1 monclient: found mon.noname-a 2023-12-15T17:02:06.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.865+0000 7fe774ac1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:06.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.865+0000 7fe774ac1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:06.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.865+0000 7fe774ac1640 1 -- 172.21.15.69:0/1968139617 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7700a8c20 con 0x7fe770060f20 2023-12-15T17:02:06.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.865+0000 7fe76f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-12-15T17:02:06.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.867+0000 7fe774ac1640 1 --2- 172.21.15.69:0/1968139617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe770060f20 0x7fe7700ad660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe768009d90 tx=0x7fe76802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2b3902ecad6037 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:06.871 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.869+0000 7fe76f7fe640 1 -- 172.21.15.69:0/1968139617 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76802cc50 con 0x7fe770060f20 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.869+0000 7fe76f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.869+0000 7fe76f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.869+0000 7fe76f7fe640 10 monclient: dump: 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.872 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.869+0000 7fe76f7fe640 10 monclient: _finish_auth 0 2023-12-15T17:02:06.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.869+0000 7fe76f7fe640 10 monclient: _check_auth_tickets 2023-12-15T17:02:06.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.869+0000 7fe76f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:06.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.869+0000 7fe776d25640 5 monclient: authenticate success, global_id 4362 2023-12-15T17:02:06.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.869+0000 7fe776d25640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:06.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.870+0000 7fe76f7fe640 1 -- 172.21.15.69:0/1968139617 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76802cdb0 con 0x7fe770060f20 2023-12-15T17:02:06.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.870+0000 7fe76f7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:06.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.870+0000 7fe76f7fe640 1 -- 172.21.15.69:0/1968139617 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768034660 con 0x7fe770060f20 2023-12-15T17:02:06.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.870+0000 7fe776d25640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.870+0000 7fe776d25640 4 set_mon_vals no callback set 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.870+0000 7fe76f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.870+0000 7fe76f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.871+0000 7fe76f7fe640 10 monclient: dump: 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:06.875 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.871+0000 7fe776d25640 10 monclient: shutdown 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.871+0000 7fe776d25640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.871+0000 7fe776d25640 1 -- 172.21.15.69:0/1968139617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe770060f20 msgr2=0x7fe7700ad660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.871+0000 7fe776d25640 1 --2- 172.21.15.69:0/1968139617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe770060f20 0x7fe7700ad660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe768009d90 tx=0x7fe76802bea0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.871+0000 7fe776d25640 1 -- 172.21.15.69:0/1968139617 shutdown_connections 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.871+0000 7fe776d25640 1 --2- 172.21.15.69:0/1968139617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe770061300 0x7fe7701430f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.871+0000 7fe776d25640 1 --2- 172.21.15.69:0/1968139617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe770060f20 0x7fe7700ad660 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.871+0000 7fe776d25640 1 -- 172.21.15.69:0/1968139617 >> 172.21.15.69:0/1968139617 conn(0x7fe7700b3e30 msgr2=0x7fe7700b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.871+0000 7fe776d25640 1 -- 172.21.15.69:0/1968139617 shutdown_connections 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.871+0000 7fe776d25640 1 -- 172.21.15.69:0/1968139617 wait complete. 2023-12-15T17:02:06.876 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.872+0000 7fe776d25640 10 monclient: build_initial_monmap 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.872+0000 7fe776d25640 10 monclient: monmap: 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.872+0000 7fe776d25640 1 librados: starting msgr at 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.872+0000 7fe776d25640 1 librados: starting objecter 2023-12-15T17:02:06.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.872+0000 7fe776d25640 1 Processor -- start 2023-12-15T17:02:06.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.872+0000 7fe776d25640 1 -- start start 2023-12-15T17:02:06.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.872+0000 7fe776d25640 1 librados: setting wanted keys 2023-12-15T17:02:06.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.872+0000 7fe776d25640 1 librados: calling monclient init 2023-12-15T17:02:06.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.872+0000 7fe776d25640 10 monclient: init 2023-12-15T17:02:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-15T17:02:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe770060f20 0x7fe770138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 10 monclient(hunting): picked mon.noname-a con 0x7fe770060f20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7700adba0 0x7fe77013f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 10 monclient(hunting): picked mon.noname-c con 0x7fe7700adba0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77013d970 0x7fe77013fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 10 monclient(hunting): picked mon.noname-b con 0x7fe77013d970 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77013f8c0 con 0x7fe770060f20 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe770138bf0 con 0x7fe77013d970 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe770140840 con 0x7fe7700adba0 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.873+0000 7fe776d25640 10 monclient(hunting): authenticate will time out at 1431.190076s 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe774ac1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe770060f20 0x7fe770138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe7752c2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77013d970 0x7fe77013fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe76ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7700adba0 0x7fe77013f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe7752c2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77013d970 0x7fe77013fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37502/0 (socket says 172.21.15.69:37502) 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe7752c2640 1 -- 172.21.15.69:0/2993444847 learned_addr learned my addr 172.21.15.69:0/2993444847 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe7752c2640 10 monclient(hunting): get_auth_request con 0x7fe77013d970 auth_method 0 2023-12-15T17:02:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe7752c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe7752c2640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe7752c2640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe774ac1640 10 monclient(hunting): get_auth_request con 0x7fe770060f20 auth_method 0 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe774ac1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe774ac1640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe774ac1640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe76ffff640 10 monclient(hunting): get_auth_request con 0x7fe7700adba0 auth_method 0 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe76ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe76ffff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe76ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe7752c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.874+0000 7fe7752c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.875+0000 7fe7752c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.875+0000 7fe774ac1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.875+0000 7fe774ac1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.875+0000 7fe774ac1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.875+0000 7fe76ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.875+0000 7fe76ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.875+0000 7fe76ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.877+0000 7fe7752c2640 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.877+0000 7fe7752c2640 1 -- 172.21.15.69:0/2993444847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7700adba0 msgr2=0x7fe77013f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.877+0000 7fe7752c2640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7700adba0 0x7fe77013f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.877+0000 7fe7752c2640 1 -- 172.21.15.69:0/2993444847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe770060f20 msgr2=0x7fe770138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.877+0000 7fe7752c2640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe770060f20 0x7fe770138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.877+0000 7fe7752c2640 10 monclient: _finish_hunting 0 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.877+0000 7fe7752c2640 1 monclient: found mon.noname-b 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.877+0000 7fe7752c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.877+0000 7fe7752c2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.877+0000 7fe7752c2640 1 -- 172.21.15.69:0/2993444847 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768009a70 con 0x7fe77013d970 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.879+0000 7fe7752c2640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77013d970 0x7fe77013fce0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe75800c430 tx=0x7fe75800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.879+0000 7fe76d7fa640 10 client.?.objecter ms_handle_connect 0x7fe77013d970 2023-12-15T17:02:06.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.879+0000 7fe76d7fa640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 1 -- 172.21.15.69:0/2993444847 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7580074e0 con 0x7fe77013d970 2023-12-15T17:02:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:02:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 10 monclient: dump: 2023-12-15T17:02:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 10 monclient: _finish_auth 0 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 10 monclient: _check_auth_tickets 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 1 -- 172.21.15.69:0/2993444847 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe758007660 con 0x7fe77013d970 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 1 -- 172.21.15.69:0/2993444847 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758004a20 con 0x7fe77013d970 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76d7fa640 10 monclient: dump: 2023-12-15T17:02:06.887 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe776d25640 5 monclient: authenticate success, global_id 4381 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe776d25640 10 monclient: _renew_subs 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe76effd640 4 set_mon_vals no callback set 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe776d25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe776d25640 1 -- 172.21.15.69:0/2993444847 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe770140220 con 0x7fe77013d970 2023-12-15T17:02:06.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe776d25640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:06.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe776d25640 10 monclient: _renew_subs 2023-12-15T17:02:06.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe776d25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:06.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe776d25640 1 -- 172.21.15.69:0/2993444847 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7701d83b0 con 0x7fe77013d970 2023-12-15T17:02:06.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.883+0000 7fe776d25640 1 librados: init done 2023-12-15T17:02:06.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.884+0000 7fe776d25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:06.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.884+0000 7fe776d25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:06.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.884+0000 7fe776d25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:02:06.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:06.884+0000 7fe776d25640 1 -- 172.21.15.69:0/2993444847 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77013f5b0 con 0x7fe77013d970 2023-12-15T17:02:07.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.124+0000 7fe76d7fa640 1 -- 172.21.15.69:0/2993444847 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe758004b80 con 0x7fe77013d970 2023-12-15T17:02:07.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.124+0000 7fe76d7fa640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe75006cba0 0x7fe75006f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:07.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.125+0000 7fe774ac1640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe75006cba0 0x7fe75006f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:07.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.125+0000 7fe774ac1640 10 monclient: get_auth_request con 0x7fe75006cba0 auth_method 0 2023-12-15T17:02:07.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.125+0000 7fe774ac1640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe75006cba0 0x7fe75006f060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe768036000 tx=0x7fe768030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:07.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.125+0000 7fe76d7fa640 10 client.4381.objecter ms_handle_connect 0x7fe75006cba0 2023-12-15T17:02:07.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.127+0000 7fe76d7fa640 1 -- 172.21.15.69:0/2993444847 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe758058940 con 0x7fe77013d970 2023-12-15T17:02:07.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.127+0000 7fe76d7fa640 10 client.4381.objecter ms_dispatch 0x7fe770000b90 osd_map(23..23 src has 1..23) v4 2023-12-15T17:02:07.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.127+0000 7fe76d7fa640 3 client.4381.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:02:07.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.127+0000 7fe76d7fa640 3 client.4381.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:02:07.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.128+0000 7fe76d7fa640 20 client.4381.objecter dump_active .. 0 homeless 2023-12-15T17:02:07.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.139+0000 7fe76d7fa640 1 -- 172.21.15.69:0/2993444847 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe758054c80 con 0x7fe77013d970 2023-12-15T17:02:07.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.139+0000 7fe76d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:07.141 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.139+0000 7fe76d7fa640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:07.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.305+0000 7fe776d25640 1 -- 172.21.15.69:0/2993444847 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe770140a00 con 0x7fe75006cba0 2023-12-15T17:02:07.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.308+0000 7fe76d7fa640 1 -- 172.21.15.69:0/2993444847 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48069 (secure 0 0 0) 0x7fe770140a00 con 0x7fe75006cba0 2023-12-15T17:02:07.310 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:02:07.311 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 10 librados: watch_flush enter 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 10 librados: watch_flush exit 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 20 client.4381.objecter shutdown clearing up homeless session... 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 10 client.4381.objecter successfully canceled tick 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 1 -- 172.21.15.69:0/2993444847 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe75006cba0 msgr2=0x7fe75006f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe75006cba0 0x7fe75006f060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe768036000 tx=0x7fe768030000 comp rx=0 tx=0).stop 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 10 monclient: shutdown 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 1 -- 172.21.15.69:0/2993444847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77013d970 msgr2=0x7fe77013fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77013d970 0x7fe77013fce0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe75800c430 tx=0x7fe75800c8f0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 1 -- 172.21.15.69:0/2993444847 shutdown_connections 2023-12-15T17:02:07.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe75006cba0 0x7fe75006f060 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77013d970 0x7fe77013fce0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7700adba0 0x7fe77013f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 1 --2- 172.21.15.69:0/2993444847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe770060f20 0x7fe770138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 1 -- 172.21.15.69:0/2993444847 >> 172.21.15.69:0/2993444847 conn(0x7fe7700b3e30 msgr2=0x7fe770071240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:07.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.310+0000 7fe776d25640 1 -- 172.21.15.69:0/2993444847 shutdown_connections 2023-12-15T17:02:07.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.311+0000 7fe776d25640 1 -- 172.21.15.69:0/2993444847 wait complete. 2023-12-15T17:02:07.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.311+0000 7fe776d25640 1 librados: shutdown 2023-12-15T17:02:07.323 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-12-15T17:02:05.438770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":111,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218876,"kb_used_data":2524,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755844,"statfs":{"total":773094113280,"available":772869984256,"internally_reserved":0,"allocated":2584576,"data_stored":1401002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":119,"apply_latency_ms":119,"commit_latency_ns":119000000,"apply_latency_ns":119000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":12,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000947"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-15T17:02:01.308854+0000","last_change":"2023-12-15T17:02:01.284924+0000","last_active":"2023-12-15T17:02:01.308854+0000","last_peered":"2023-12-15T17:02:01.308854+0000","last_clean":"2023-12-15T17:02:01.308854+0000","last_became_active":"2023-12-15T17:02:01.275938+0000","last_became_peered":"2023-12-15T17:02:01.275938+0000","last_unstale":"2023-12-15T17:02:01.308854+0000","last_undegraded":"2023-12-15T17:02:01.308854+0000","last_fullsized":"2023-12-15T17:02:01.308854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:35:22.642595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-15T17:02:03.938406+0000","last_change":"2023-12-15T17:01:59.949551+0000","last_active":"2023-12-15T17:02:03.938406+0000","last_peered":"2023-12-15T17:02:03.938406+0000","last_clean":"2023-12-15T17:02:03.938406+0000","last_became_active":"2023-12-15T17:01:59.918026+0000","last_became_peered":"2023-12-15T17:01:59.918026+0000","last_unstale":"2023-12-15T17:02:03.938406+0000","last_undegraded":"2023-12-15T17:02:03.938406+0000","last_fullsized":"2023-12-15T17:02:03.938406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:10:51.112278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-15T17:02:01.408317+0000","last_change":"2023-12-15T17:02:01.380902+0000","last_active":"2023-12-15T17:02:01.408317+0000","last_peered":"2023-12-15T17:02:01.408317+0000","last_clean":"2023-12-15T17:02:01.408317+0000","last_became_active":"2023-12-15T17:02:01.375672+0000","last_became_peered":"2023-12-15T17:02:01.375672+0000","last_unstale":"2023-12-15T17:02:01.408317+0000","last_undegraded":"2023-12-15T17:02:01.408317+0000","last_fullsized":"2023-12-15T17:02:01.408317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:15:10.893330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-15T17:02:04.067403+0000","last_change":"2023-12-15T17:01:59.938931+0000","last_active":"2023-12-15T17:02:04.067403+0000","last_peered":"2023-12-15T17:02:04.067403+0000","last_clean":"2023-12-15T17:02:04.067403+0000","last_became_active":"2023-12-15T17:01:59.908680+0000","last_became_peered":"2023-12-15T17:01:59.908680+0000","last_unstale":"2023-12-15T17:02:04.067403+0000","last_undegraded":"2023-12-15T17:02:04.067403+0000","last_fullsized":"2023-12-15T17:02:04.067403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:35:37.366491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-15T17:02:01.718405+0000","last_change":"2023-12-15T17:02:01.718405+0000","last_active":"2023-12-15T17:02:01.718405+0000","last_peered":"2023-12-15T17:02:01.718405+0000","last_clean":"2023-12-15T17:02:01.718405+0000","last_became_active":"2023-12-15T17:02:01.713188+0000","last_became_peered":"2023-12-15T17:02:01.713188+0000","last_unstale":"2023-12-15T17:02:01.718405+0000","last_undegraded":"2023-12-15T17:02:01.718405+0000","last_fullsized":"2023-12-15T17:02:01.718405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:56:55.902253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-15T17:02:03.965576+0000","last_change":"2023-12-15T17:01:59.559261+0000","last_active":"2023-12-15T17:02:03.965576+0000","last_peered":"2023-12-15T17:02:03.965576+0000","last_clean":"2023-12-15T17:02:03.965576+0000","last_became_active":"2023-12-15T17:01:59.531480+0000","last_became_peered":"2023-12-15T17:01:59.531480+0000","last_unstale":"2023-12-15T17:02:03.965576+0000","last_undegraded":"2023-12-15T17:02:03.965576+0000","last_fullsized":"2023-12-15T17:02:03.965576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:51:07.527384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-15T17:02:01.306086+0000","last_change":"2023-12-15T17:02:01.240306+0000","last_active":"2023-12-15T17:02:01.306086+0000","last_peered":"2023-12-15T17:02:01.306086+0000","last_clean":"2023-12-15T17:02:01.306086+0000","last_became_active":"2023-12-15T17:02:01.208353+0000","last_became_peered":"2023-12-15T17:02:01.208353+0000","last_unstale":"2023-12-15T17:02:01.306086+0000","last_undegraded":"2023-12-15T17:02:01.306086+0000","last_fullsized":"2023-12-15T17:02:01.306086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T00:32:59.159368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-15T17:02:01.625426+0000","last_change":"2023-12-15T17:02:01.625426+0000","last_active":"2023-12-15T17:02:01.625426+0000","last_peered":"2023-12-15T17:02:01.625426+0000","last_clean":"2023-12-15T17:02:01.625426+0000","last_became_active":"2023-12-15T17:02:01.616517+0000","last_became_peered":"2023-12-15T17:02:01.616517+0000","last_unstale":"2023-12-15T17:02:01.625426+0000","last_undegraded":"2023-12-15T17:02:01.625426+0000","last_fullsized":"2023-12-15T17:02:01.625426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:10:18.201816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'75","reported_seq":118,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-15T17:02:02.580906+0000","last_change":"2023-12-15T17:01:52.907228+0000","last_active":"2023-12-15T17:02:02.580906+0000","last_peered":"2023-12-15T17:02:02.580906+0000","last_clean":"2023-12-15T17:02:02.580906+0000","last_became_active":"2023-12-15T17:01:52.878062+0000","last_became_peered":"2023-12-15T17:01:52.878062+0000","last_unstale":"2023-12-15T17:02:02.580906+0000","last_undegraded":"2023-12-15T17:02:02.580906+0000","last_fullsized":"2023-12-15T17:02:02.580906+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_clean_scrub_stamp":"2023-12-15T17:01:50.252516+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:47:14.717362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":111,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":111,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":74820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0850000000000009}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1180000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.824999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.739999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.972000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.884}]}]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":610304,"data_stored":467250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.6600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.105}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.127000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.715999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.614000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.98}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.339}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.076000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.282}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":52.637999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7449999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.155999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":35.234000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.366}]}]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":69410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.554}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.507000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.728000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.441000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.773}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.808}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.048}]}]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":80568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6589999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.327000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8629999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6890000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.048999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.289000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.186}]}]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":80568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.801}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.823}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.814}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.448}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.641999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.673999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.606}]}]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":634880,"data_stored":478408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9619999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9580000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5519999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.282}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.952000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.558999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":80568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1429999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9749999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.827}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.817}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.305}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.532}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-15T17:02:07.324 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-15T17:02:07.325 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-15T17:02:07.325 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-15T17:02:07.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-15T17:02:07.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.420+0000 7fe678925640 10 monclient: get_monmap_and_config 2023-12-15T17:02:07.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.420+0000 7fe678925640 10 monclient: build_initial_monmap 2023-12-15T17:02:07.422 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.420+0000 7fe678925640 10 monclient: monmap: 2023-12-15T17:02:07.422 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:07.422 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:02:07.422169+0000 2023-12-15T17:02:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:02:07.422169+0000 2023-12-15T17:02:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:02:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:02:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:02:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:02:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:02:07.423 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.421+0000 7fe678925640 1 Processor -- start 2023-12-15T17:02:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.421+0000 7fe678925640 1 -- start start 2023-12-15T17:02:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.421+0000 7fe678925640 10 monclient: init 2023-12-15T17:02:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.421+0000 7fe678925640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.421+0000 7fe678925640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-12-15T17:02:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.421+0000 7fe678925640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe67406ccd0 0x7fe67413f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe678925640 10 monclient(hunting): picked mon.noname-c con 0x7fe67406ccd0 addr v2:172.21.15.192:3300/0 2023-12-15T17:02:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe678925640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe67406c0e0 0x7fe67413f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe678925640 10 monclient(hunting): picked mon.noname-a con 0x7fe67406c0e0 addr v2:172.21.15.69:3300/0 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe678925640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe67406d0b0 addr v1:172.21.15.69:6789/0 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe678925640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6740afb80 con 0x7fe67406d0b0 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe678925640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe678925640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6740b0010 con 0x7fe67406c0e0 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe678925640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe678925640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe674071070 con 0x7fe67406ccd0 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe678925640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe678925640 10 monclient(hunting): authenticate will time out at 1431.738415s 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe671d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe67406c0e0 0x7fe67413f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe67259c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe67406ccd0 0x7fe67413f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe67259c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe67406ccd0 0x7fe67413f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:37518/0 (socket says 172.21.15.69:37518) 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe67259c640 1 -- 172.21.15.69:0/1585791751 learned_addr learned my addr 172.21.15.69:0/1585791751 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe67259c640 10 monclient(hunting): get_auth_request con 0x7fe67406ccd0 auth_method 0 2023-12-15T17:02:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe67259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe67259c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.422+0000 7fe67259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.423+0000 7fe671d9b640 10 monclient(hunting): get_auth_request con 0x7fe67406c0e0 auth_method 0 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.423+0000 7fe671d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.423+0000 7fe671d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.423+0000 7fe671d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.423+0000 7fe67259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.423+0000 7fe67259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.423+0000 7fe67259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.423+0000 7fe671d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.423+0000 7fe671d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.423+0000 7fe671d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.424+0000 7fe67259c640 10 monclient(hunting): handle_auth_done global_id 4384 payload 274 2023-12-15T17:02:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.424+0000 7fe67259c640 1 -- 172.21.15.69:0/1585791751 >> v2:172.21.15.69:3300/0 conn(0x7fe67406c0e0 msgr2=0x7fe67413f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:02:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.424+0000 7fe67259c640 1 --2- 172.21.15.69:0/1585791751 >> v2:172.21.15.69:3300/0 conn(0x7fe67406c0e0 0x7fe67413f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.424+0000 7fe67259c640 1 -- 172.21.15.69:0/1585791751 >> v1:172.21.15.69:6789/0 conn(0x7fe67406d0b0 legacy=0x7fe674071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.424+0000 7fe67259c640 10 monclient: _finish_hunting 0 2023-12-15T17:02:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.424+0000 7fe67259c640 1 monclient: found mon.noname-c 2023-12-15T17:02:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.424+0000 7fe67259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.424+0000 7fe67259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:02:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.424+0000 7fe67259c640 1 -- 172.21.15.69:0/1585791751 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6740712e0 con 0x7fe67406ccd0 2023-12-15T17:02:07.428 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.426+0000 7fe67259c640 1 --2- 172.21.15.69:0/1585791751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67406ccd0 0x7fe67413f110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe6740b21f0 tx=0x7fe66402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11b01d0b3bc59b9f server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 1 -- 172.21.15.69:0/1585791751 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe664031870 con 0x7fe67406ccd0 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 10 monclient: dump: 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:07.432 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 10 monclient: _finish_auth 0 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 10 monclient: _check_auth_tickets 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 1 -- 172.21.15.69:0/1585791751 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6640319d0 con 0x7fe67406ccd0 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 5 monclient: authenticate success, global_id 4384 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 10 monclient: get_monmap_and_config success 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 1 -- 172.21.15.69:0/1585791751 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe664031d60 con 0x7fe67406ccd0 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 4 set_mon_vals no callback set 2023-12-15T17:02:07.433 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe67159a640 10 monclient: dump: 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:07.434 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 10 monclient: shutdown 2023-12-15T17:02:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 1 -- 172.21.15.69:0/1585791751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67406ccd0 msgr2=0x7fe67413f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 1 --2- 172.21.15.69:0/1585791751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67406ccd0 0x7fe67413f110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe6740b21f0 tx=0x7fe66402b910 comp rx=0 tx=0).stop 2023-12-15T17:02:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 1 -- 172.21.15.69:0/1585791751 shutdown_connections 2023-12-15T17:02:07.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 1 --2- 172.21.15.69:0/1585791751 >> v2:172.21.15.69:3300/0 conn(0x7fe67406c0e0 0x7fe67413f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 1 --2- 172.21.15.69:0/1585791751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67406ccd0 0x7fe67413f110 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.430+0000 7fe678925640 1 -- 172.21.15.69:0/1585791751 >> 172.21.15.69:0/1585791751 conn(0x7fe6740b5ca0 msgr2=0x7fe6740b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.431+0000 7fe678925640 1 -- 172.21.15.69:0/1585791751 shutdown_connections 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.431+0000 7fe678925640 1 -- 172.21.15.69:0/1585791751 wait complete. 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.431+0000 7fe678925640 10 monclient: build_initial_monmap 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.431+0000 7fe678925640 10 monclient: monmap: 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:02:07.438 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 librados: starting msgr at 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 librados: starting objecter 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 Processor -- start 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 -- start start 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 librados: setting wanted keys 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 librados: calling monclient init 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 10 monclient: init 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 10 monclient: _reopen_session rank -1 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67406c0e0 0x7fe67413ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 10 monclient(hunting): picked mon.noname-b con 0x7fe67406c0e0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:02:07.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe67406ccd0 0x7fe674135bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 10 monclient(hunting): picked mon.noname-c con 0x7fe67406ccd0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe67406d0b0 0x7fe674136110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 10 monclient(hunting): picked mon.noname-a con 0x7fe67406d0b0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6741382f0 con 0x7fe67406d0b0 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6740ad8a0 con 0x7fe67406c0e0 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 10 monclient(hunting): start opening mon connection 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6741387d0 con 0x7fe67406ccd0 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 10 monclient(hunting): _renew_subs 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.432+0000 7fe678925640 10 monclient(hunting): authenticate will time out at 1431.749077s 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe672d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe67406d0b0 0x7fe674136110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe671d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe67406ccd0 0x7fe674135bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe67259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67406c0e0 0x7fe67413ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe672d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe67406d0b0 0x7fe674136110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58874/0 (socket says 172.21.15.69:58874) 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe672d9d640 1 -- 172.21.15.69:0/842946794 learned_addr learned my addr 172.21.15.69:0/842946794 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe672d9d640 10 monclient(hunting): get_auth_request con 0x7fe67406d0b0 auth_method 0 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe672d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe672d9d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe672d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe67259c640 10 monclient(hunting): get_auth_request con 0x7fe67406c0e0 auth_method 0 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe67259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe67259c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe67259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe671d9b640 10 monclient(hunting): get_auth_request con 0x7fe67406ccd0 auth_method 0 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe671d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe671d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.433+0000 7fe671d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.434+0000 7fe672d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.434+0000 7fe672d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.434+0000 7fe672d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.434+0000 7fe67259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.434+0000 7fe67259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.434+0000 7fe67259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.434+0000 7fe671d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:02:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.434+0000 7fe671d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:02:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.434+0000 7fe671d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:02:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.436+0000 7fe672d9d640 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2023-12-15T17:02:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.436+0000 7fe672d9d640 1 -- 172.21.15.69:0/842946794 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe67406ccd0 msgr2=0x7fe674135bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.436+0000 7fe672d9d640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe67406ccd0 0x7fe674135bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.436+0000 7fe672d9d640 1 -- 172.21.15.69:0/842946794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67406c0e0 msgr2=0x7fe67413ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.436+0000 7fe672d9d640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67406c0e0 0x7fe67413ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.436+0000 7fe672d9d640 10 monclient: _finish_hunting 0 2023-12-15T17:02:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.436+0000 7fe672d9d640 1 monclient: found mon.noname-a 2023-12-15T17:02:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.436+0000 7fe672d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:02:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.436+0000 7fe672d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:02:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.436+0000 7fe672d9d640 1 -- 172.21.15.69:0/842946794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6640094d0 con 0x7fe67406d0b0 2023-12-15T17:02:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.439+0000 7fe672d9d640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe67406d0b0 0x7fe674136110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c6b0 tx=0x7fe66c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.439+0000 7fe662ffd640 10 client.?.objecter ms_handle_connect 0x7fe67406d0b0 2023-12-15T17:02:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.439+0000 7fe662ffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:02:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 1 -- 172.21.15.69:0/842946794 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c003b70 con 0x7fe67406d0b0 2023-12-15T17:02:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:02:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 10 monclient: dump: 2023-12-15T17:02:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 10 monclient: _finish_auth 0 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 1 -- 172.21.15.69:0/842946794 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c003cf0 con 0x7fe67406d0b0 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 1 -- 172.21.15.69:0/842946794 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c003ea0 con 0x7fe67406d0b0 2023-12-15T17:02:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe678925640 5 monclient: authenticate success, global_id 4371 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe678925640 10 monclient: _renew_subs 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe678925640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe678925640 1 -- 172.21.15.69:0/842946794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe674138a50 con 0x7fe67406d0b0 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe662ffd640 10 monclient: dump: 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:02:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:02:07.448 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:02:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe670d99640 4 set_mon_vals no callback set 2023-12-15T17:02:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe678925640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:02:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe678925640 10 monclient: _renew_subs 2023-12-15T17:02:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe678925640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe678925640 1 -- 172.21.15.69:0/842946794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6740aca90 con 0x7fe67406d0b0 2023-12-15T17:02:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.443+0000 7fe678925640 1 librados: init done 2023-12-15T17:02:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.444+0000 7fe678925640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.444+0000 7fe678925640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:07.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.444+0000 7fe678925640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:07.449 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.444+0000 7fe678925640 1 -- 172.21.15.69:0/842946794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6741384f0 con 0x7fe67406d0b0 2023-12-15T17:02:07.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.696+0000 7fe662ffd640 1 -- 172.21.15.69:0/842946794 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe66c005310 con 0x7fe67406d0b0 2023-12-15T17:02:07.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.696+0000 7fe662ffd640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe65406cba0 0x7fe65406f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:02:07.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.697+0000 7fe67259c640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe65406cba0 0x7fe65406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:02:07.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.697+0000 7fe67259c640 10 monclient: get_auth_request con 0x7fe65406cba0 auth_method 0 2023-12-15T17:02:07.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.697+0000 7fe662ffd640 10 client.4371.objecter ms_handle_connect 0x7fe65406cba0 2023-12-15T17:02:07.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.697+0000 7fe67259c640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe65406cba0 0x7fe65406f060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe664036000 tx=0x7fe66403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:02:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.711+0000 7fe662ffd640 1 -- 172.21.15.69:0/842946794 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe66c058a50 con 0x7fe67406d0b0 2023-12-15T17:02:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.711+0000 7fe662ffd640 10 client.4371.objecter ms_dispatch 0x7fe674000b90 osd_map(23..23 src has 1..23) v4 2023-12-15T17:02:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.711+0000 7fe662ffd640 3 client.4371.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:02:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.711+0000 7fe662ffd640 3 client.4371.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:02:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.711+0000 7fe662ffd640 20 client.4371.objecter dump_active .. 0 homeless 2023-12-15T17:02:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.711+0000 7fe662ffd640 1 -- 172.21.15.69:0/842946794 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe66c054d90 con 0x7fe67406d0b0 2023-12-15T17:02:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.711+0000 7fe662ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:02:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.711+0000 7fe662ffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:02:07.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.936+0000 7fe678925640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-12-15T17:02:07.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.936+0000 7fe678925640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-12-15T17:02:07.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.936+0000 7fe678925640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:02:07.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.936+0000 7fe678925640 1 -- 172.21.15.69:0/842946794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe6741384f0 con 0x7fe67406d0b0 2023-12-15T17:02:07.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.941+0000 7fe662ffd640 1 -- 172.21.15.69:0/842946794 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe6741384f0 con 0x7fe67406d0b0 2023-12-15T17:02:07.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.941+0000 7fe662ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-12-15T17:02:07.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.941+0000 7fe662ffd640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:02:07.943 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:02:07.943 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-15T17:02:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 10 librados: watch_flush enter 2023-12-15T17:02:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 10 librados: watch_flush exit 2023-12-15T17:02:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 20 client.4371.objecter shutdown clearing up homeless session... 2023-12-15T17:02:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 10 client.4371.objecter successfully canceled tick 2023-12-15T17:02:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 1 -- 172.21.15.69:0/842946794 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe65406cba0 msgr2=0x7fe65406f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe65406cba0 0x7fe65406f060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe664036000 tx=0x7fe66403c040 comp rx=0 tx=0).stop 2023-12-15T17:02:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 10 monclient: shutdown 2023-12-15T17:02:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:02:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 1 -- 172.21.15.69:0/842946794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe67406d0b0 msgr2=0x7fe674136110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:02:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe67406d0b0 0x7fe674136110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c6b0 tx=0x7fe66c00cb70 comp rx=0 tx=0).stop 2023-12-15T17:02:07.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 1 -- 172.21.15.69:0/842946794 shutdown_connections 2023-12-15T17:02:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe65406cba0 0x7fe65406f060 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe67406d0b0 0x7fe674136110 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe67406ccd0 0x7fe674135bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 1 --2- 172.21.15.69:0/842946794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67406c0e0 0x7fe67413ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:02:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.943+0000 7fe678925640 1 -- 172.21.15.69:0/842946794 >> 172.21.15.69:0/842946794 conn(0x7fe6740b5ca0 msgr2=0x7fe674073c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:02:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.944+0000 7fe678925640 1 -- 172.21.15.69:0/842946794 shutdown_connections 2023-12-15T17:02:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.944+0000 7fe678925640 1 -- 172.21.15.69:0/842946794 wait complete. 2023-12-15T17:02:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:02:07.944+0000 7fe678925640 1 librados: shutdown 2023-12-15T17:02:07.956 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-15T17:02:07.956 INFO:teuthology.run_tasks:Running task workunit... 2023-12-15T17:02:07.969 INFO:tasks.workunit:Pulling workunits from ref 021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-15T17:02:07.969 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-15T17:02:07.969 DEBUG:teuthology.orchestra.run.smithi069:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-15T17:02:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T17:02:07.984 INFO:teuthology.orchestra.run.smithi069.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-15T17:02:07.984 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-15T17:02:08.039 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-15T17:02:08.039 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-15T17:02:08.094 INFO:tasks.workunit:timeout=6h 2023-12-15T17:02:08.094 INFO:tasks.workunit:cleanup=True 2023-12-15T17:02:08.095 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-15T17:02:08.149 INFO:tasks.workunit.client.0.smithi069.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-15T17:02:20.115 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:30.060 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:20.119 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:30.064 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:20.157 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:30.102 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:20.167 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:30.106 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:20.168 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:30.113 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:20.170 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:30.116 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:20.174 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:30.113 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:20.211 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:30.156 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:20.213 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:30.153 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:40.117 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:50.062 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:40.170 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:50.110 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:40.171 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:50.111 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:40.181 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:50.127 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:40.188 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:50.127 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:40.201 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:50.146 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:40.228 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:50.173 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:40.231 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:50.177 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:02:40.249 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:50.194 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:00.180 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:10.125 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:00.225 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:10.165 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:00.226 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:10.166 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:00.239 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:10.178 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:00.257 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:10.203 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:00.287 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:10.232 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:00.330 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:10.275 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:00.338 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:10.283 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:00.353 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:10.299 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:20.153 INFO:tasks.workunit.client.0.smithi069.stderr:Note: switching to '021ac1670a21c19759080c6da90baaeb42e7d175'. 2023-12-15T17:03:20.153 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr:state without impacting any branches by switching back to a branch. 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr: git switch -c 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr:Or undo this operation with: 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr: git switch - 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-12-15T17:03:20.154 INFO:tasks.workunit.client.0.smithi069.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-15T17:03:20.155 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-12-15T17:03:20.155 INFO:tasks.workunit.client.0.smithi069.stderr:HEAD is now at 021ac1670a2 Merge branch 'wip-63713' of https://github.com/vshankar/ceph into wip-yuri10-testing-2023-12-12-1229 2023-12-15T17:03:20.156 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-15T17:03:20.219 INFO:tasks.workunit.client.0.smithi069.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-15T17:03:20.221 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-15T17:03:20.221 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-15T17:03:20.238 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:30.184 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:20.285 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:30.230 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:20.288 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:30.227 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:20.290 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:30.229 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:20.305 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:30.245 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:20.310 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:30.256 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:20.339 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:30.284 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:20.341 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:30.286 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:20.355 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:30.300 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:20.858 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-15T17:03:20.915 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-15T17:03:20.957 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-15T17:03:20.958 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-15T17:03:20.959 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-15T17:03:20.999 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-15T17:03:21.002 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-15T17:03:21.003 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-15T17:03:21.057 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-12-15T17:03:21.059 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-12-15T17:03:21.059 DEBUG:teuthology.orchestra.run.smithi069:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=021ac1670a21c19759080c6da90baaeb42e7d175 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-12-15T17:03:21.117 INFO:tasks.workunit.client.0.smithi069.stderr:+ parallel=1 2023-12-15T17:03:21.117 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' '' = --serial ']' 2023-12-15T17:03:21.117 INFO:tasks.workunit.client.0.smithi069.stderr:+ color= 2023-12-15T17:03:21.117 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -t 1 ']' 2023-12-15T17:03:21.117 INFO:tasks.workunit.client.0.smithi069.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-12-15T17:03:21.117 INFO:tasks.workunit.client.0.smithi069.stderr:+ GTEST_OUTPUT=xml:/home/ubuntu/cephtest/archive/gtest_xml_report 2023-12-15T17:03:21.117 INFO:tasks.workunit.client.0.smithi069.stderr:+ declare -A pids 2023-12-15T17:03:21.118 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.118 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_aio 2023-12-15T17:03:21.118 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_aio' 2023-12-15T17:03:21.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_aio 2023-12-15T17:03:21.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.118 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_aio 2023-12-15T17:03:21.119 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91942 2023-12-15T17:03:21.119 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_aio on pid 91942' 2023-12-15T17:03:21.119 INFO:tasks.workunit.client.0.smithi069.stdout:test api_aio on pid 91942 2023-12-15T17:03:21.122 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91942 2023-12-15T17:03:21.122 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.122 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.122 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-12-15T17:03:21.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_aio_pp 2023-12-15T17:03:21.122 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_aio_pp' 2023-12-15T17:03:21.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_aio_pp 2023-12-15T17:03:21.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.122 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.122 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.124 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_aio_pp 2023-12-15T17:03:21.124 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91949 2023-12-15T17:03:21.124 INFO:tasks.workunit.client.0.smithi069.stdout:test api_aio_pp on pid 91949 2023-12-15T17:03:21.127 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_aio_pp on pid 91949' 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91949 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_io 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_io' 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_io 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.128 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.129 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.130 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.131 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.131 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.131 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.131 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.131 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.131 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.131 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_io 2023-12-15T17:03:21.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91958 2023-12-15T17:03:21.131 INFO:tasks.workunit.client.0.smithi069.stdout:test api_io on pid 91958 2023-12-15T17:03:21.136 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_io on pid 91958' 2023-12-15T17:03:21.136 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91958 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_io_pp 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_io_pp' 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_io_pp 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.137 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.141 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.141 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.141 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_io_pp 2023-12-15T17:03:21.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91971 2023-12-15T17:03:21.141 INFO:tasks.workunit.client.0.smithi069.stdout:test api_io_pp on pid 91971 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_io_pp on pid 91971' 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91971 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_asio 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_asio' 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_asio 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.147 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.149 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.150 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.151 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_asio 2023-12-15T17:03:21.152 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91986 2023-12-15T17:03:21.153 INFO:tasks.workunit.client.0.smithi069.stdout:test api_asio on pid 91986 2023-12-15T17:03:21.159 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_asio on pid 91986' 2023-12-15T17:03:21.159 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=91986 2023-12-15T17:03:21.159 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.159 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.159 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-12-15T17:03:21.159 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_list 2023-12-15T17:03:21.159 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_list' 2023-12-15T17:03:21.159 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.159 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_list 2023-12-15T17:03:21.159 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.159 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.160 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.161 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.162 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.163 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91942/exe 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.165 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.166 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.166 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.166 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.166 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.166 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_list 2023-12-15T17:03:21.166 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.166 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92001 2023-12-15T17:03:21.166 INFO:tasks.workunit.client.0.smithi069.stdout:test api_list on pid 92001 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_list on pid 92001' 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92001 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_lock 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_lock' 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_lock 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.174 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.175 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.176 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.177 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_aio: /' 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.178 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_aio.log 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.179 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.180 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91949/exe 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.181 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.182 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.182 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.182 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.182 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.182 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.182 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_lock 2023-12-15T17:03:21.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92023 2023-12-15T17:03:21.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_lock on pid 92023' 2023-12-15T17:03:21.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-12-15T17:03:21.182 INFO:tasks.workunit.client.0.smithi069.stdout:test api_lock on pid 92023 2023-12-15T17:03:21.192 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92023 2023-12-15T17:03:21.192 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.192 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.192 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.192 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.192 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.192 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.192 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.192 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.192 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.192 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_lock_pp 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.193 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.194 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.195 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_lock_pp' 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-12-15T17:03:21.196 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_aio_pp: /' 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_lock_pp 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.197 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.198 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91958/exe 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.200 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_lock_pp 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.201 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92043 2023-12-15T17:03:21.202 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.202 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.202 INFO:tasks.workunit.client.0.smithi069.stdout:test api_lock_pp on pid 92043 2023-12-15T17:03:21.209 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_lock_pp on pid 92043' 2023-12-15T17:03:21.209 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.209 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92043 2023-12-15T17:03:21.209 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.209 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.209 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.209 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.209 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.210 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91971/exe 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_io.log 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_misc 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.211 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_misc' 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_io: /' 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.212 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.213 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.214 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_misc 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.215 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.216 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.216 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.216 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.216 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2023-12-15T17:03:21.216 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.216 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.216 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.216 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.216 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_misc 2023-12-15T17:03:21.216 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.216 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92067 2023-12-15T17:03:21.217 INFO:tasks.workunit.client.0.smithi069.stdout:test api_misc on pid 92067 2023-12-15T17:03:21.227 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_misc on pid 92067' 2023-12-15T17:03:21.227 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92067 2023-12-15T17:03:21.227 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.227 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.227 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.228 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.229 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_misc_pp 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_misc_pp' 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_io_pp: /' 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/91986/exe 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.232 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.233 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2023-12-15T17:03:21.234 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.235 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_asio.log 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.236 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92001/exe 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_misc_pp 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.237 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.238 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.238 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_misc_pp 2023-12-15T17:03:21.238 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92092 2023-12-15T17:03:21.238 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_misc_pp on pid 92092' 2023-12-15T17:03:21.238 INFO:tasks.workunit.client.0.smithi069.stdout:test api_misc_pp on pid 92092 2023-12-15T17:03:21.252 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92092 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_tier_pp 2023-12-15T17:03:21.253 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_tier_pp' 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.254 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_list.log 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.255 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.257 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.258 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.259 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.260 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92043/exe 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_list: /' 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.261 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-12-15T17:03:21.262 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.263 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_tier_pp 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92023/exe 2023-12-15T17:03:21.264 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_tier_pp 2023-12-15T17:03:21.265 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92119 2023-12-15T17:03:21.265 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.265 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.265 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.265 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.265 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.265 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.265 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_tier_pp on pid 92119' 2023-12-15T17:03:21.265 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.265 INFO:tasks.workunit.client.0.smithi069.stdout:test api_tier_pp on pid 92119 2023-12-15T17:03:21.270 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.270 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92119 2023-12-15T17:03:21.270 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.270 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2023-12-15T17:03:21.270 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.270 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.270 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_asio: /' 2023-12-15T17:03:21.270 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.270 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.270 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.271 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_lock.log 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_lock_pp: /' 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_pool 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_pool' 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2023-12-15T17:03:21.272 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_pool 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_lock: /' 2023-12-15T17:03:21.273 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_pool 2023-12-15T17:03:21.274 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92154 2023-12-15T17:03:21.274 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_pool on pid 92154' 2023-12-15T17:03:21.274 INFO:tasks.workunit.client.0.smithi069.stdout:test api_pool on pid 92154 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92154 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_snapshots 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_snapshots' 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.284 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.285 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_snapshots 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.287 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.288 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92092/exe 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.289 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.290 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.291 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.292 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.293 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_misc_pp: /' 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_snapshots 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92196 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_snapshots on pid 92196' 2023-12-15T17:03:21.294 INFO:tasks.workunit.client.0.smithi069.stdout:test api_snapshots on pid 92196 2023-12-15T17:03:21.298 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92196 2023-12-15T17:03:21.298 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.298 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.298 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-12-15T17:03:21.298 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.299 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.300 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_snapshots_pp 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_snapshots_pp' 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.301 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.302 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_snapshots_pp 2023-12-15T17:03:21.302 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92067/exe 2023-12-15T17:03:21.302 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_snapshots_pp 2023-12-15T17:03:21.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92207 2023-12-15T17:03:21.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_snapshots_pp on pid 92207' 2023-12-15T17:03:21.302 INFO:tasks.workunit.client.0.smithi069.stdout:test api_snapshots_pp on pid 92207 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92207 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_stat 2023-12-15T17:03:21.311 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_stat' 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_misc.log 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_misc: /' 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.312 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_stat 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.313 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.314 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.315 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92154/exe 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.316 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.317 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.318 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.318 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.318 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.318 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.318 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.318 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_stat 2023-12-15T17:03:21.318 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.318 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.318 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92224 2023-12-15T17:03:21.318 INFO:tasks.workunit.client.0.smithi069.stdout:test api_stat on pid 92224 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_stat on pid 92224' 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92224 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.324 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.325 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_stat_pp 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_stat_pp' 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.326 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_stat_pp 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_pool.log 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_pool: /' 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.328 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.328 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.328 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.328 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_stat_pp 2023-12-15T17:03:21.328 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.328 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.328 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92252 2023-12-15T17:03:21.328 INFO:tasks.workunit.client.0.smithi069.stdout:test api_stat_pp on pid 92252 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_stat_pp on pid 92252' 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92252 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.332 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] Global test environment set-up. 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] 12 tests from AsioRados 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.227+0000 7f540ee3b780 10 monclient: get_monmap_and_config 2023-12-15T17:03:21.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.227+0000 7f540ee3b780 10 monclient: build_initial_monmap 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.227+0000 7f540ee3b780 10 monclient: monmap: 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 0 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 0.000000 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 0.000000 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 0 (unknown) 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.227+0000 7f540ee3b780 1 Processor -- start 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.227+0000 7f540ee3b780 1 -- start start 2023-12-15T17:03:21.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.227+0000 7f540ee3b780 10 monclient: init 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 10 monclient: _reopen_session rank -1 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c5ae9ffe00 0x55c5aea8c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 10 monclient(hunting): picked mon.noname-a con 0x55c5ae9ffe00 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c5aea8cb70 0x55c5aeb37a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 10 monclient(hunting): picked mon.noname-b con 0x55c5aea8cb70 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c5aea898e0 0x55c5aeb37f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 10 monclient(hunting): picked mon.noname-c con 0x55c5aea898e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 10 monclient(hunting): start opening mon connection 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ae975b50 con 0x55c5ae9ffe00 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 10 monclient(hunting): start opening mon connection 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ae90f970 con 0x55c5aea8cb70 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 10 monclient(hunting): start opening mon connection 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5ae90f610 con 0x55c5aea898e0 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 10 monclient(hunting): _renew_subs 2023-12-15T17:03:21.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.228+0000 7f540ee3b780 10 monclient(hunting): authenticate will time out at 1505.544213s 2023-12-15T17:03:21.341 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.341 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.341 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.341 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.341 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.341 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.341 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.342 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.343 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_watch_notify 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_watch_notify' 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_watch_notify 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92119/exe 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.344 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.345 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.346 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.231+0000 7f540c10e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c5ae9ffe00 0x55c5aea8c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.231+0000 7f540c10e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c5ae9ffe00 0x55c5aea8c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48226/0 (socket says 172.21.15.69:48226) 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.231+0000 7f540c10e640 1 -- 172.21.15.69:0/1154995264 learned_addr learned my addr 172.21.15.69:0/1154995264 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.231+0000 7f540c10e640 10 monclient(hunting): get_auth_request con 0x55c5ae9ffe00 auth_method 0 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.231+0000 7f540c10e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.231+0000 7f540c10e640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.231+0000 7f540c10e640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.232+0000 7f540c10e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.232+0000 7f540c10e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.232+0000 7f540c10e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.238+0000 7f540c90f640 1 --2- 172.21.15.69:0/1154995264 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c5aea898e0 0x55c5aeb37f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.240+0000 7f540c10e640 10 monclient(hunting): handle_auth_done global_id 4386 payload 274 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.240+0000 7f540c10e640 1 -- 172.21.15.69:0/1154995264 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c5aea898e0 msgr2=0x55c5aeb37f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.240+0000 7f540c10e640 1 --2- 172.21.15.69:0/1154995264 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c5aea898e0 0x55c5aeb37f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:03:21.347 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.240+0000 7f540c10e640 1 -- 172.21.15.69:0/1154995264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c5aea8cb70 msgr2=0x55c5aeb37a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:03:21.348 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.240+0000 7f540c10e640 1 --2- 172.21.15.69:0/1154995264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c5aea8cb70 0x55c5aeb37a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:03:21.348 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.240+0000 7f540c10e640 10 monclient: _finish_hunting 0 2023-12-15T17:03:21.348 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.240+0000 7f540c10e640 1 monclient: found mon.noname-a 2023-12-15T17:03:21.348 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.240+0000 7f540c10e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:03:21.348 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.240+0000 7f540c10e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:03:21.348 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.240+0000 7f540c10e640 1 -- 172.21.15.69:0/1154995264 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc005de0 con 0x55c5ae9ffe00 2023-12-15T17:03:21.348 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_watch_notify 2023-12-15T17:03:21.348 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92309 2023-12-15T17:03:21.349 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_watch_notify on pid 92309' 2023-12-15T17:03:21.349 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.349 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.244+0000 7f540c10e640 1 --2- 172.21.15.69:0/1154995264 >> [v2:172.21.15.69:test api_watch_notify on pid 92309 2023-12-15T17:03:21.356 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92309 2023-12-15T17:03:21.356 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.356 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.357 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_tier_pp: /' 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.358 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.359 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_watch_notify_pp 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_watch_notify_pp' 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.360 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_watch_notify_pp 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.361 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.362 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.363 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.363 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.363 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.363 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.363 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.363 INFO:tasks.workunit.client.0.smithi069.stdout:3300/0,v1:172.21.15.69:6789/0] conn(0x55c5ae9ffe00 0x55c5aea8c0c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0219a0 tx=0x7f53fc050070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b86db51f736b997 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:03:21.363 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 1 -- 172.21.15.69:0/1154995264 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc056910 con 0x55c5ae9ffe00 2023-12-15T17:03:21.363 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:03:21.363 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:03:21.363 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 10 monclient: dump: 2023-12-15T17:03:21.363 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 19 (squid) 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 10 monclient: _finish_auth 0 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 10 monclient: _check_auth_tickets 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 1 -- 172.21.15.69:0/1154995264 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc056a70 con 0x55c5ae9ffe00 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 1 -- 172.21.15.69:0/1154995264 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc056e00 con 0x55c5ae9ffe00 2023-12-15T17:03:21.364 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540a90b640 10 monclient: dump: 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 19 (squid) 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540ee3b780 5 monclient: authenticate success, global_id 4386 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540ee3b780 10 monclient: get_monmap_and_config success 2023-12-15T17:03:21.365 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.255+0000 7f540ee3b780 4 set_mon_vals no callback set 2023-12-15T17:03:21.366 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.256+0000 7f540ee3b780 10 monclient: shutdown 2023-12-15T17:03:21.366 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.256+0000 7f540ee3b780 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:03:21.366 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_watch_notify_pp 2023-12-15T17:03:21.366 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92339 2023-12-15T17:03:21.366 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-12-15T17:03:21.366 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-12-15T17:03:21.366 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_watch_notify_pp on pid 92339' 2023-12-15T17:03:21.367 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.256+0000 7f540ee3b780 1 -- 172.21.15.69:0/1154995264 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c5ae9ffe00 msgr2=0x55c5aea8c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_downtest api_watch_notify_pp on pid 92339 2023-12-15T17:03:21.367 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92339 2023-12-15T17:03:21.368 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.368 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.368 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.368 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.368 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.368 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.368 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.368 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.368 INFO:tasks.workunit.client.0.smithi069.stdout: 2023-12-15T17:03:21.368 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.256+0000 7f540ee3b780 1 --2- 172.21.15.69:0/1154995264 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c5ae9ffe00 0x55c5aea8c0c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0219a0 tx=0x7f53fc050070 comp rx=0 tx=0).stop 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.257+0000 7f540ee3b780 1 -- 172.21.15.69:0/1154995264 shutdown_connections 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.257+0000 7f540ee3b780 1 --2- 172.21.15.69:0/1154995264 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c5aea898e0 0x55c5aeb37f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.257+0000 7f540ee3b780 1 --2- 172.21.15.69:0/1154995264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c5aea8cb70 0x55c5aeb37a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.257+0000 7f540ee3b780 1 --2- 172.21.15.69:0/1154995264 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c5ae9ffe00 0x55c5aea8c0c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.257+0000 7f540ee3b780 1 -- 172.21.15.69:0/1154995264 >> 172.21.15.69:0/1154995264 conn(0x55c5ae9b9ee0 msgr2=0x55c5aea891d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.257+0000 7f540ee3b780 1 -- 172.21.15.69:0/1154995264 shutdown_connections 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.257+0000 7f540ee3b780 1 -- 172.21.15.69:0/1154995264 wait complete. 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.257+0000 7f540ee3b780 10 monclient: build_initial_monmap 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.257+0000 7f540ee3b780 10 monclient: monmap: 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 0 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 0.000000 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 0.000000 2023-12-15T17:03:21.369 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 0 (unknown) 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.258+0000 7f540ee3b780 1 librados: starting msgr at 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.258+0000 7f540ee3b780 1 librados: starting objecter 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.258+0000 7f540ee3b780 1 Processor -- start 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.258+0000 7f540ee3b780 1 -- start start 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.258+0000 7f540ee3b780 1 librados: setting wanted keys 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.258+0000 7f540ee3b780 1 librados: calling monclient init 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.258+0000 7f540ee3b780 10 monclient: init 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 10 monclient: _reopen_session rank -1 2023-12-15T17:03:21.370 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 10 monclient: _add_conns ranks=[0,2,1] 2023-12-15T17:03:21.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c5ae9ffe00 0x55c5ae9fdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:03:21.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 10 monclient(hunting): picked mon.noname-a con 0x55c5ae9ffe00 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:03:21.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c5aea898e0 0x55c5ae9d46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:03:21.382 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.382 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.382 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.382 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.382 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.382 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.382 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.382 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.382 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.383 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.384 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.385 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cmd 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cmd' 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.386 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cmd 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.387 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.388 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.389 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.390 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92196/exe 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.391 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2023-12-15T17:03:21.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-12-15T17:03:21.393 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_snapshots: /' 2023-12-15T17:03:21.393 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cmd 2023-12-15T17:03:21.393 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92374 2023-12-15T17:03:21.393 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259test api_cmd on pid 92374 2023-12-15T17:03:21.393 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cmd on pid 92374' 2023-12-15T17:03:21.394 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92374 2023-12-15T17:03:21.394 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.394 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.394 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.394 INFO:tasks.workunit.client.0.smithi069.stdout:+0000 7f540ee3b780 10 monclient(hunting): picked mon.noname-c con 0x55c5aea898e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:03:21.394 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c5aea8cb70 0x55c5aea87310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:03:21.394 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 10 monclient(hunting): picked mon.noname-b con 0x55c5aea8cb70 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:03:21.394 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 10 monclient(hunting): start opening mon connection 2023-12-15T17:03:21.394 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ae8ef4e0 con 0x55c5ae9ffe00 2023-12-15T17:03:21.394 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 10 monclient(hunting): start opening mon connection 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5aea8c4c0 con 0x55c5aea8cb70 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 10 monclient(hunting): start opening mon connection 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5ae9bd160 con 0x55c5aea898e0 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 10 monclient(hunting): _renew_subs 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.259+0000 7f540ee3b780 10 monclient(hunting): authenticate will time out at 1505.574685s 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.268+0000 7f540b90d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c5aea898e0 0x55c5ae9d46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.268+0000 7f540b90d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c5aea898e0 0x55c5ae9d46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:56446/0 (socket says 172.21.15.69:56446) 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.268+0000 7f540b90d640 1 -- 172.21.15.69:0/642570505 learned_addr learned my addr 172.21.15.69:0/642570505 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.268+0000 7f540b90d640 10 monclient(hunting): get_auth_request con 0x55c5aea898e0 auth_method 0 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.268+0000 7f540b90d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.268+0000 7f540b90d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.268+0000 7f540b90d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.269+0000 7f540b90d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.269+0000 7f540b90d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.269+0000 7f540b90d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:03:21.395 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.274+0000 7f540b90d640 10 monclient(hunting): handle_auth_done global_id 4379 payload 824 2023-12-15T17:03:21.396 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.274+0000 7f540b90d640 1 -- 172.21.15.69:0/642570505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c5aea8cb70 msgr2=0x55c5aea87310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:03:21.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cmd_pp 2023-12-15T17:03:21.405 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.405 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.405 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cmd_pp' 2023-12-15T17:03:21.405 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cmd_pp 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.406 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.408 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.409 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92224/exe 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.410 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_stat.log 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.411 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_stat: /' 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.414 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.414 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cmd_pp 2023-12-15T17:03:21.414 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.414 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.414 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92398 2023-12-15T17:03:21.414 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.274+0000 7f540b90d640 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c5aea8cb70 0x55c5aea87310 unknown :-1test api_cmd_pp on pid 92398 2023-12-15T17:03:21.415 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cmd_pp on pid 92398' 2023-12-15T17:03:21.415 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92398 2023-12-15T17:03:21.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.415 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.415 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.416 INFO:tasks.workunit.client.0.smithi069.stdout: s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.274+0000 7f540b90d640 1 -- 172.21.15.69:0/642570505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c5ae9ffe00 msgr2=0x55c5ae9fdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.274+0000 7f540b90d640 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c5ae9ffe00 0x55c5ae9fdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.274+0000 7f540b90d640 10 monclient: _finish_hunting 0 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.274+0000 7f540b90d640 1 monclient: found mon.noname-c 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.274+0000 7f540b90d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.274+0000 7f540b90d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.274+0000 7f540b90d640 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5ae935760 con 0x55c5aea898e0 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.280+0000 7f540b90d640 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c5aea898e0 0x55c5ae9d46d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5404008090 tx=0x7f5404008550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.280+0000 7f5409909640 10 client.?.objecter ms_handle_connect 0x55c5aea898e0 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.280+0000 7f5409909640 10 client.?.objecter resend_mon_ops 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404010640 con 0x55c5aea898e0 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:03:21.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 10 monclient: dump: 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 19 (squid) 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 10 monclient: _finish_auth 0 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 10 monclient: _check_auth_tickets 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54040107a0 con 0x55c5aea898e0 2023-12-15T17:03:21.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:03:21.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404010b50 con 0x55c5aea898e0 2023-12-15T17:03:21.426 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.426 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.426 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.426 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.427 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_service 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_service' 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.428 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.429 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_service 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.430 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.432 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.432 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.432 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.432 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.432 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.433 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92252/exe 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_service 2023-12-15T17:03:21.434 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92438 2023-12-15T17:03:21.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 10 monclient: handle_monmap mon_map magic: 0 test api_service on pid 92438 2023-12-15T17:03:21.442 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_service on pid 92438' 2023-12-15T17:03:21.442 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92438 2023-12-15T17:03:21.442 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.442 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.442 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-12-15T17:03:21.442 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.442 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_service_pp 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_service_pp' 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92339/exe 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.443 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.444 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_service_pp 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.445 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.446 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.447 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_service_pp 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92465 2023-12-15T17:03:21.448 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.449 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.449 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_service_pp on pid 92465' 2023-12-15T17:03:21.449 INFO:tasks.workunit.client.0.smithi069.stdout:test api_service_pp on pid 92465 2023-12-15T17:03:21.463 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.463 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-12-15T17:03:21.463 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92465 2023-12-15T17:03:21.463 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.463 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.463 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.463 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92207/exe 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.464 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.465 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.466 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92309/exe 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.467 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-12-15T17:03:21.468 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_stat_pp: /' 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_c_write_operations 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_c_write_operations' 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.470 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_c_write_operations 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_watch_notify: /' 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.474 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.476 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.476 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.476 INFO:tasks.workunit.client.0.smithi069.stdout:test api_c_write_operations on pid 92499 2023-12-15T17:03:21.476 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_c_write_operations 2023-12-15T17:03:21.476 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92499 2023-12-15T17:03:21.476 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_c_write_operations on pid 92499' 2023-12-15T17:03:21.476 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92499 2023-12-15T17:03:21.476 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stdout:v1 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f5409909640 10 monclient: dump: 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 19 (squid) 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:03:21.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f540ee3b780 5 monclient: authenticate success, global_id 4379 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f540ee3b780 10 monclient: _renew_subs 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f540ee3b780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c5ae91b770 con 0x55c5aea898e0 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.285+0000 7f540ee3b780 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.286+0000 7f540ee3b780 10 monclient: _renew_subs 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.286+0000 7f540ee3b780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.286+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5ae90e920 con 0x55c5aea898e0 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.286+0000 7f540ee3b780 1 librados: init done 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.286+0000 7f540ee3b780 10 librados: wait_for_osdmap waiting 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.286+0000 7f540a90b640 4 set_mon_vals no callback set 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.320+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f5404010cb0 con 0x55c5aea898e0 2023-12-15T17:03:21.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.321+0000 7f5409909640 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f53f806adc0 0x7f53f806d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:03:21.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.322+0000 7f540c10e640 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f53f806adc0 0x7f53f806d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:03:21.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.323+0000 7f540c10e640 10 monclient: get_auth_request con 0x7f53f806adc0 auth_method 0 2023-12-15T17:03:21.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.324+0000 7f5409909640 10 client.4379.objecter ms_handle_connect 0x7f53f806adc0 2023-12-15T17:03:21.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.324+0000 7f540c10e640 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f53f806adc0 0x7f53f806d280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f53fc059040 tx=0x7f53fc005ac0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:03:21.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.333+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f54040577b0 con 0x55c5aea898e0 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92374/exe 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.495 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.496 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.497 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.498 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.499 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.500 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_c_read_operations 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_c_read_operations' 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.501 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.502 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cmd.log 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cmd: /' 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.503 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92438/exe 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.504 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.505 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_service.log 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.506 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_c_read_operations 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_service: /' 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.507 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.508 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_c_read_operations 2023-12-15T17:03:21.509 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92556 2023-12-15T17:03:21.509 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_c_read_operations on pid 92556' 2023-12-15T17:03:21.509 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:test api_c_read_operations on pid 92556 2023-12-15T17:03:21.514 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92556 2023-12-15T17:03:21.514 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.514 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.514 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.514 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.514 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.514 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cls_remote_reads 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cls_remote_reads' 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cls_remote_reads 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92398/exe 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.517 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cls_remote_reads 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92579 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-12-15T17:03:21.518 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cls_remote_reads on pid 92579' 2023-12-15T17:03:21.519 INFO:tasks.workunit.client.0.smithi069.stdout:test api_cls_remote_reads on pid 92579 2023-12-15T17:03:21.531 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92579 2023-12-15T17:03:21.531 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s list_parallel 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' list_parallel' 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92465/exe 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.533 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.534 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.535 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2023-12-15T17:03:21.536 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.537 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.538 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92499/exe 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.539 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2023-12-15T17:03:21.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_service_pp: /' 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo list_parallel 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=list_parallel 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92632 2023-12-15T17:03:21.542 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test list_parallel on pid 92632' 2023-12-15T17:03:21.543 INFO:tasks.workunit.client.0.smithi069.stdout:test list_parallel on pid 92632 2023-12-15T17:03:21.556 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92632 2023-12-15T17:03:21.556 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.556 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.556 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s open_pools_parallel 2023-12-15T17:03:21.556 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.556 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' open_pools_parallel' 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.557 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.558 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.559 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.560 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.561 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92579/exe 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo open_pools_parallel 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92556/exe 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=open_pools_parallel 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92688 2023-12-15T17:03:21.567 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test open_pools_parallel on pid 92688' 2023-12-15T17:03:21.568 INFO:tasks.workunit.client.0.smithi069.stdout:test open_pools_parallel on pid 92688 2023-12-15T17:03:21.572 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92688 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s delete_pools_parallel 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' delete_pools_parallel' 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.573 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo delete_pools_parallel 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.574 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.575 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.576 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=delete_pools_parallel 2023-12-15T17:03:21.577 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92706 2023-12-15T17:03:21.577 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test delete_pools_parallel on pid 92706' 2023-12-15T17:03:21.577 INFO:tasks.workunit.client.0.smithi069.stdout:test delete_pools_parallel on pid 92706 2023-12-15T17:03:21.596 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=92706 2023-12-15T17:03:21.596 INFO:tasks.workunit.client.0.smithi069.stderr:+ ret=0 2023-12-15T17:03:21.596 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-12-15T17:03:21.596 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:03:21.596 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92119 2023-12-15T17:03:21.596 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92119 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.597 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.598 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-15T17:03:21.599 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.600 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.601 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92632/exe 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.602 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.603 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.604 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ list_parallel: /' 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_list_parallel.log 2023-12-15T17:03:21.605 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-15T17:03:21.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.607 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92688/exe 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.608 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2023-12-15T17:03:21.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-15T17:03:21.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2023-12-15T17:03:21.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/92706/exe 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-12-15T17:03:21.612 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-12-15T17:03:22.781 INFO:tasks.workunit.client.0.smithi069.stdout:03:21.333+0000 7f5409909640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_map(23..23 src has 1..23) v4 2023-12-15T17:03:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.333+0000 7f5409909640 3 client.4379.objecter handle_osd_map got epochs [23,23] > 0 2023-12-15T17:03:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.333+0000 7f5409909640 3 client.4379.objecter handle_osd_map decoding full epoch 23 2023-12-15T17:03:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.333+0000 7f5409909640 20 client.4379.objecter dump_active .. 0 homeless 2023-12-15T17:03:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.333+0000 7f540ee3b780 10 librados: wait_for_osdmap done waiting 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.333+0000 7f540ee3b780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.333+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55c5ae96e9e0 con 0x55c5aea898e0 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.343+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5404056e80 con 0x55c5aea898e0 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.343+0000 7f5409909640 10 monclient: handle_get_version_reply finishing 1 version 23 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.343+0000 7f540a10a640 10 client.4379.objecter _get_latest_version latest 23, have it 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.343+0000 7f540ee3b780 10 client.4379.objecter create_pool name=ceph_test_rados_api_asio 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.343+0000 7f540ee3b780 10 client.4379.objecter pool_op_submit 1 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.343+0000 7f540ee3b780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:21.343+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55c5ae97fbe0 con 0x55c5aea898e0 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.774+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== mon.2 v2:172.21.15.69:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v24) v1 ==== 43+0+0 (secure 0 0 0) 0x7f5404056bf0 con 0x55c5aea898e0 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.774+0000 7f5409909640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 pool_op_reply(tid 1 (0) Success v24) v1 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.774+0000 7f5409909640 10 client.4379.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v24) v1 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.774+0000 7f5409909640 10 client.4379.objecter have request 1 at 0x55c5ae9fe6f0 Op: create 2023-12-15T17:03:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.774+0000 7f5409909640 20 client.4379.objecter waiting for client to reach epoch 24 before calling back 2023-12-15T17:03:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.774+0000 7f5409909640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:03:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.774+0000 7f5409909640 10 monclient: _renew_subs 2023-12-15T17:03:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.774+0000 7f5409909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:03:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.774+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=24}) v3 -- 0x7f53f8070b00 con 0x55c5aea898e0 2023-12-15T17:03:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.774+0000 7f5409909640 10 client.4379.objecter done 2023-12-15T17:03:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== mon.2 v2:172.21.15.69:3301/0 8 ==== osd_map(24..24 src has 1..24) v4 ==== 2115+0+0 (secure 0 0 0) 0x7f54040570b0 con 0x55c5aea898e0 2023-12-15T17:03:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f5409909640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_map(24..24 src has 1..24) v4 2023-12-15T17:03:25.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f5409909640 3 client.4379.objecter handle_osd_map got epochs [24,24] > 23 2023-12-15T17:03:25.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f5409909640 3 client.4379.objecter handle_osd_map decoding incremental epoch 24 2023-12-15T17:03:25.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f5409909640 20 client.4379.objecter dump_active .. 0 homeless 2023-12-15T17:03:25.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 10 librados: set snap read head -> 1 2023-12-15T17:03:25.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 10 librados: writefull oid=exist nspace= 2023-12-15T17:03:25.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5aea87850 2023-12-15T17:03:25.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 24 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:25.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:25.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:25.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x55c5aea87cf0 0x55c5aea72860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 2 osd.2 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 2 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 15 client.4379.objecter _send_op 2 to 7.1 on osd.2 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:2 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) v8 -- 0x55c5aea72da0 con 0x55c5aea87cf0 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.780+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.895+0000 7f540c90f640 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x55c5aea87cf0 0x55c5aea72860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.918+0000 7f540c90f640 10 monclient: get_auth_request con 0x55c5aea87cf0 auth_method 0 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.978+0000 7f540c90f640 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x55c5aea87cf0 0x55c5aea72860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:22.978+0000 7f5409909640 10 client.4379.objecter ms_handle_connect 0x55c5aea87cf0 2023-12-15T17:03:25.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:24.259+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:25.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:24.259+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:25.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:24.259+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:25.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:24.259+0000 7f5408907640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:03:29.539 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:25.259+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:25.259+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:25.259+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.259+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.259+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.259+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.286+0000 7f540e372640 10 client.4379.objecter tick 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.775+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 1 ==== osd_map(25..25 src has 1..25) v4 ==== 9084+0+0 (crc 0 0 0) 0x7f5400005a70 con 0x55c5aea87cf0 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.775+0000 7f5409909640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_map(25..25 src has 1..25) v4 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.775+0000 7f5409909640 3 client.4379.objecter handle_osd_map got epochs [25,25] > 24 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.775+0000 7f5409909640 3 client.4379.objecter handle_osd_map decoding incremental epoch 25 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.775+0000 7f5409909640 10 client.4379.objecter checking op 2 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.775+0000 7f5409909640 20 client.4379.objecter _calc_target epoch 25 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:29.540 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.775+0000 7f5409909640 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.775+0000 7f5409909640 20 client.4379.objecter dump_active .. 0 homeless 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:26.775+0000 7f5409909640 20 client.4379.objecter 2 7.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:27.258+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:27.258+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:27.258+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:28.258+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:28.258+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:28.258+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.258+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.258+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.258+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 2 ==== osd_op_reply(2 exist [writefull 0~5] v25'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f540000c520 con 0x55c5aea87cf0 2023-12-15T17:03:29.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(2 exist [writefull 0~5] v25'1 uv1 ondisk = 0) v8 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 2 ondisk uv 1 in 7.1 attempt 0 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 2 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540c90f640 15 client.4379.objecter _finish_op 2 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 2 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 10 librados: Objecter returned from writefull r=0 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5aea87850 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 25 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:29.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 3 osd.2 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 3 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 15 client.4379.objecter _send_op 3 to 7.1 on osd.2 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:3 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25) v8 -- 0x55c5aea73130 con 0x55c5aea87cf0 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 25 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 20 client.4379.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 4 osd.2 2023-12-15T17:03:29.543 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:29.544 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 4 2023-12-15T17:03:29.544 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 15 client.4379.objecter _send_op 4 to 7.1 on osd.2 2023-12-15T17:03:29.544 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:4 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25) v8 -- 0x55c5ae9cd410 con 0x55c5aea87cf0 2023-12-15T17:03:29.544 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:29.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.537+0000 7f540ee3b780 5 client.4379.objecter 2 in flight 2023-12-15T17:03:29.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (378 ms) 2023-12-15T17:03:29.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-12-15T17:03:29.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f540000c520 con 0x55c5aea87cf0 2023-12-15T17:03:29.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-15T17:03:29.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:29.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 3 ondisk uv 1 in 7.1 attempt 0 2023-12-15T17:03:29.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 5 2023-12-15T17:03:29.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 3 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 15 client.4379.objecter _finish_op 3 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 3 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 5 client.4379.objecter 1 in flight 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f540000c520 con 0x55c5aea87cf0 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 3 ==== osd_map(26..27 src has 1..27) v4 ==== 8620+0+0 (crc 0 0 0) 0x7f540000c7a0 con 0x55c5aea87cf0 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 4 ondisk uv 0 in 7.1 attempt 0 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 4 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f5409909640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_map(26..27 src has 1..27) v4 2023-12-15T17:03:29.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 15 client.4379.objecter _finish_op 4 2023-12-15T17:03:29.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:29.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 4 2023-12-15T17:03:29.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:29.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f5409909640 3 client.4379.objecter handle_osd_map got epochs [26,27] > 25 2023-12-15T17:03:29.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f5409909640 3 client.4379.objecter handle_osd_map decoding incremental epoch 26 2023-12-15T17:03:29.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f5409909640 3 client.4379.objecter handle_osd_map decoding incremental epoch 27 2023-12-15T17:03:29.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f5409909640 20 client.4379.objecter dump_active .. 0 homeless 2023-12-15T17:03:29.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:29.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 27 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:29.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:29.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:29.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:29.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 5 osd.2 2023-12-15T17:03:29.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 5 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 15 client.4379.objecter _send_op 5 to 7.1 on osd.2 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:5 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) v8 -- 0x55c5aea87850 con 0x55c5aea87cf0 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9cd7f0 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 27 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 20 client.4379.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 6 osd.2 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 6 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 15 client.4379.objecter _send_op 6 to 7.1 on osd.2 2023-12-15T17:03:29.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:6 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) v8 -- 0x55c5aeb51ec0 con 0x55c5aea87cf0 2023-12-15T17:03:29.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:29.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.916+0000 7f540ee3b780 5 client.4379.objecter 2 in flight 2023-12-15T17:03:29.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.984+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 6 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:30.028 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.984+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-15T17:03:30.028 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.984+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:30.028 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.984+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 5 ondisk uv 1 in 7.1 attempt 0 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.984+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 5 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.984+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 5 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.984+0000 7f540c90f640 15 client.4379.objecter _finish_op 5 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.984+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.984+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 5 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:29.984+0000 7f540c90f640 5 client.4379.objecter 1 in flight 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 7 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 6 ondisk uv 0 in 7.1 attempt 0 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540c90f640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 6 2023-12-15T17:03:30.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540c90f640 15 client.4379.objecter _finish_op 6 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 6 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (111 ms) 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 27 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 7 osd.2 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 7 2023-12-15T17:03:30.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 15 client.4379.objecter _send_op 7 to 7.1 on osd.2 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:7 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) v8 -- 0x55c5aeb72230 con 0x55c5aea87cf0 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.026+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5aeb925a0 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 27 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 20 client.4379.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 8 osd.2 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 8 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 15 client.4379.objecter _send_op 8 to 7.1 on osd.2 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:8 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) v8 -- 0x55c5aeb92a40 con 0x55c5aea87cf0 2023-12-15T17:03:30.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.027+0000 7f540ee3b780 5 client.4379.objecter 2 in flight 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.147+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 8 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 7 ondisk uv 1 in 7.1 attempt 0 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 5 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 7 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 15 client.4379.objecter _finish_op 7 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 7 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 5 client.4379.objecter 1 in flight 2023-12-15T17:03:30.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 9 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:30.239 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-15T17:03:30.239 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:30.239 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 8 ondisk uv 0 in 7.1 attempt 0 2023-12-15T17:03:30.239 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-15T17:03:30.239 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:30.239 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 8 2023-12-15T17:03:30.239 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 15 client.4379.objecter _finish_op 8 2023-12-15T17:03:30.239 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:30.239 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 8 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (121 ms) 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 20 librados: queue_aio_write 0x55c5ae96e700 completion 0x55c5aea8c2f0 write_seq 1 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 27 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 9 osd.2 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 9 2023-12-15T17:03:30.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 15 client.4379.objecter _send_op 9 to 7.1 on osd.2 2023-12-15T17:03:30.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:9 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) v8 -- 0x55c5ae9cd7f0 con 0x55c5aea87cf0 2023-12-15T17:03:30.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:30.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:30.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.148+0000 7f540ee3b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-15T17:03:30.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.237+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 10 ==== osd_map(28..28 src has 1..28) v4 ==== 1023+0+0 (crc 0 0 0) 0x7f540000ce10 con 0x55c5aea87cf0 2023-12-15T17:03:30.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.237+0000 7f5409909640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_map(28..28 src has 1..28) v4 2023-12-15T17:03:32.000 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.237+0000 7f5409909640 3 client.4379.objecter handle_osd_map got epochs [28,28] > 27 2023-12-15T17:03:32.000 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.237+0000 7f5409909640 3 client.4379.objecter handle_osd_map decoding incremental epoch 28 2023-12-15T17:03:32.000 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.237+0000 7f5409909640 10 client.4379.objecter checking op 9 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.237+0000 7f5409909640 20 client.4379.objecter _calc_target epoch 28 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.237+0000 7f5409909640 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.237+0000 7f5409909640 20 client.4379.objecter dump_active .. 0 homeless 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.237+0000 7f5409909640 20 client.4379.objecter 9 7.d3a59201 osd.2 exist [write 0~5 in=5b] 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.258+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.258+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:30.258+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.258+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.258+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.258+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.286+0000 7f540e372640 10 client.4379.objecter tick 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (1850 ms) 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-12-15T17:03:32.001 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 11 ==== osd_op_reply(9 exist [write 0~5] v28'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(9 exist [write 0~5] v28'2 uv2 ondisk = 0) v8 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 9 ondisk uv 2 in 7.1 attempt 0 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 9 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540c90f640 15 client.4379.objecter _finish_op 9 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 9 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540c90f640 20 librados: complete_aio_write 0x55c5aea8c2f0 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 20 librados: queue_aio_write 0x55c5ae96e700 completion 0x55c5ae9cdb50 write_seq 2 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 28 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:32.002 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:32.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 10 client.4379. api_service: [==========] Running 4 tests from 1 test suite. 2023-12-15T17:03:32.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] Global test environment set-up. 2023-12-15T17:03:32.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] 4 tests from LibRadosService 2023-12-15T17:03:32.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-12-15T17:03:32.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5350 ms) 2023-12-15T17:03:32.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-12-15T17:03:32.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.RegisterLate (186 ms) 2023-12-15T17:03:32.333 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_WARN 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 5 pool(s) do not have an application enabled 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: x(active, since 2m), standbys: y 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 97s), 8 in (since 2m) 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2023-12-15T17:03:32.334 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 26 pools, 777 pgs 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 2 objects, 449 KiB 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 98.842% pgs unknown 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 768 unknown 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 9 active+clean 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_WARN 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 8 pool(s) do not have an application enabled 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-15T17:03:32.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: x(active, since 2m), standbys: y 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 99s), 8 in (since 2m) 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: foo: 4 portals active (1 hosts, 3 zones) 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 27 pools, 785 pgs 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 3 objects, 449 KiB 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 37.325% pgs unknown 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 60.382% pgs not active 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 415 creating+peering 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 293 unknown 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 59 creating+activating 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 18 active+clean 2023-12-15T17:03:32.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: io: 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: client: 207 B/s wr, 0 op/s rd, 0 op/s wr 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_WARN 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 9 pool(s) do not have an application enabled 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: x(active, since 2m), standbys: y 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 102s), 8 in (since 2m) 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-12-15T17:03:32.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-12-15T17:03:32.338 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:32.338 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2023-12-15T17:03:32.338 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 30 pools, 881 pgs 2023-12-15T17:03:32.338 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 3 objects, 449 KiB 2023-12-15T17:03:32.338 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-15T17:03:35.032 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 44.154% pgs unkobjecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:35.032 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 10 osd.2 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 10 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 15 client.4379.objecter _send_op 10 to 7.1 on osd.2 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:10 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) v8 -- 0x55c5ae9cd7f0 con 0x55c5aea87cf0 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:31.997+0000 7f540ee3b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.064+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 12 ==== osd_map(29..29 src has 1..29) v4 ==== 1380+0+0 (crc 0 0 0) 0x7f540000c9b0 con 0x55c5aea87cf0 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.064+0000 7f5409909640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_map(29..29 src has 1..29) v4 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.064+0000 7f5409909640 3 client.4379.objecter handle_osd_map got epochs [29,29] > 28 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.064+0000 7f5409909640 3 client.4379.objecter handle_osd_map decoding incremental epoch 29 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.064+0000 7f5409909640 10 client.4379.objecter checking op 10 2023-12-15T17:03:35.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.064+0000 7f5409909640 20 client.4379.objecter _calc_target epoch 29 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.064+0000 7f5409909640 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.064+0000 7f5409909640 20 client.4379.objecter dump_active .. 0 homeless 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.064+0000 7f5409909640 20 client.4379.objecter 10 7.d3a59201 osd.2 exist [write 0~5 in=5b] 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.259+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.259+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:32.259+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:33.259+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:33.259+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:33.259+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:34.259+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:34.259+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:34.259+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:35.034 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:34.259+0000 7f5408907640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:03:35.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.030+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 13 ==== osd_op_reply(10 exist [write 0~5] v29'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:35.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.030+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(10 exist [write 0~5] v29'3 uv3 ondisk = 0) v8 2023-12-15T17:03:35.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.030+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:35.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.030+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 10 ondisk uv 3 in 7.1 attempt 0 2023-12-15T17:03:35.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.030+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:35.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.030+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 10 2023-12-15T17:03:35.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.030+0000 7f540c90f640 15 client.4379.objecter _finish_op 10 2023-12-15T17:03:35.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.030+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.030+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 10 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.030+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.030+0000 7f540c90f640 20 librados: complete_aio_write 0x55c5ae9cdb50 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3033 ms) 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 20 librados: queue_aio_write 0x55c5ae96e700 completion 0x55c5ae9cda10 write_seq 3 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 29 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 11 osd.2 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:35.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 11 2023-12-15T17:03:35.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 15 client.4379.objecter _send_op 11 to 7.1 on osd.2 2023-12-15T17:03:35.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:11 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) v8 -- 0x55c5aeb553c0 con 0x55c5aea87cf0 2023-12-15T17:03:35.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:35.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:35.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.031+0000 7f540ee3b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-15T17:03:35.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.073+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 14 ==== osd_map(30..30 src has 1..30) v4 ==== 992+0+0 (crc 0 0 0) 0x7f540000cb70 con 0x55c5aea87cf0 2023-12-15T17:03:35.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.073+0000 7f5409909640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_map(30..30 src has 1..30) v4 2023-12-15T17:03:36.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.073+0000 7f5409909640 3 client.4379.objecter handle_osd_map got epochs [30,30] > 29 2023-12-15T17:03:36.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.073+0000 7f5409909640 3 client.4379.objecter handle_osd_map decoding incremental epoch 30 2023-12-15T17:03:36.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.073+0000 7f5409909640 10 client.4379.objecter checking op 11 2023-12-15T17:03:36.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.073+0000 7f5409909640 20 client.4379.objecter _calc_target epoch 30 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:36.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.073+0000 7f5409909640 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:36.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.073+0000 7f5409909640 20 client.4379.objecter dump_active .. 0 homeless 2023-12-15T17:03:36.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.073+0000 7f5409909640 20 client.4379.objecter 11 7.d3a59201 osd.2 exist [write 0~5 in=5b] 2023-12-15T17:03:36.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.259+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:36.534 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.259+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:35.259+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.259+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.259+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.259+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.287+0000 7f540e372640 10 client.4379.objecter tick 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 15 ==== osd_op_reply(11 exist [write 0~5] v30'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(11 exist [write 0~5] v30'4 uv4 ondisk = 0) v8 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 11 ondisk uv 4 in 7.1 attempt 0 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 11 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540c90f640 15 client.4379.objecter _finish_op 11 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 11 2023-12-15T17:03:36.535 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:36.536 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540c90f640 20 librados: complete_aio_write 0x55c5ae9cda10 2023-12-15T17:03:36.536 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1501 ms) 2023-12-15T17:03:36.536 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-12-15T17:03:36.536 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:36.536 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 30 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:36.536 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:36.536 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:36.536 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:36.608 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 12 osd.2 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 12 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 15 client.4379.objecter _send_op 12 to 7.1 on osd.2 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:12 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c5aeb54d50 con 0x55c5aea87cf0 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5aeb551c0 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 30 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 20 client.4379.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.609 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 13 osd.2 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 13 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 15 client.4379.objecter _send_op 13 to 7.1 on osd.2 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:13 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c5aeb52220 con 0x55c5aea87cf0 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.532+0000 7f540ee3b780 5 client.4379.objecter 2 in flight 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.604+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 16 ==== osd_map(31..31 src has 1..31) v4 ==== 1007+0+0 (crc 0 0 0) 0x7f5400019850 con 0x55c5aea87cf0 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.604+0000 7f5409909640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_map(31..31 src has 1..31) v4 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.604+0000 7f5409909640 3 client.4379.objecter handle_osd_map got epochs [31,31] > 30 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.604+0000 7f5409909640 3 client.4379.objecter handle_osd_map decoding incremental epoch 31 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.605+0000 7f5409909640 10 client.4379.objecter checking op 12 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.605+0000 7f5409909640 20 client.4379.objecter _calc_target epoch 31 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.605+0000 7f5409909640 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:36.610 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.605+0000 7f5409909640 10 client.4379.objecter checking op 13 2023-12-15T17:03:36.661 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.605+0000 7f5409909640 20 client.4379.objecter _calc_target epoch 31 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:36.661 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.605+0000 7f5409909640 20 client.4379.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2023-12-15T17:03:36.661 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.605+0000 7f5409909640 20 client.4379.objecter dump_active .. 0 homeless 2023-12-15T17:03:36.661 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.605+0000 7f5409909640 20 client.4379.objecter 12 7.d3a59201 osd.2 exist [read 0~0] 2023-12-15T17:03:36.661 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.605+0000 7f5409909640 20 client.4379.objecter 13 7.a7bbaa1 osd.2 noexist [read 0~0] 2023-12-15T17:03:36.661 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 17 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:36.661 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-15T17:03:36.661 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:36.661 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 12 ondisk uv 4 in 7.1 attempt 0 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 5 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 12 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 15 client.4379.objecter _finish_op 12 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 12 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 5 client.4379.objecter 1 in flight 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 18 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 13 ondisk uv 0 in 7.1 attempt 0 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 13 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 15 client.4379.objecter _finish_op 13 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.662 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 13 2023-12-15T17:03:36.663 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:36.663 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (127 ms) 2023-12-15T17:03:36.663 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-12-15T17:03:36.663 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:36.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 31 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:36.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:36.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:36.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:36.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 14 osd.2 2023-12-15T17:03:36.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:36.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 14 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 15 client.4379.objecter _send_op 14 to 7.1 on osd.2 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:14 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) v8 -- 0x55c5aeb551c0 con 0x55c5aea87cf0 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5aeb52220 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 31 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 20 client.4379.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 15 osd.2 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 15 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 15 client.4379.objecter _send_op 15 to 7.1 on osd.2 2023-12-15T17:03:36.688 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:15 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) v8 -- 0x55c5aeb526c0 con 0x55c5aea87cf0 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.659+0000 7f540ee3b780 5 client.4379.objecter 2 in flight 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 19 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 14 ondisk uv 4 in 7.1 attempt 0 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 5 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 14 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 15 client.4379.objecter _finish_op 14 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 14 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 5 client.4379.objecter 1 in flight 2023-12-15T17:03:36.689 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 20 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 15 ondisk uv 0 in 7.1 attempt 0 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 15 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 15 client.4379.objecter _finish_op 15 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 15 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (27 ms) 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 31 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:36.690 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:36.691 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:36.691 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:36.691 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 16 osd.2 2023-12-15T17:03:36.691 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:36.691 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 16 2023-12-15T17:03:36.691 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 15 client.4379.objecter _send_op 16 to 7.1 on osd.2 2023-12-15T17:03:36.691 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:16 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) v8 -- 0x55c5aeb551c0 con 0x55c5aea87cf0 2023-12-15T17:03:36.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:36.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5aeb52220 2023-12-15T17:03:36.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 31 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2023-12-15T17:03:36.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 20 client.4379.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2023-12-15T17:03:36.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:36.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 17 osd.2 2023-12-15T17:03:36.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 17 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 15 client.4379.objecter _send_op 17 to 7.1 on osd.2 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:17 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) v8 -- 0x55c5aeb526c0 con 0x55c5aea87cf0 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.685+0000 7f540ee3b780 5 client.4379.objecter 2 in flight 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.695+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 21 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.695+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.695+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.695+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 16 ondisk uv 4 in 7.1 attempt 0 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.695+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 5 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.695+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 16 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.695+0000 7f540c90f640 15 client.4379.objecter _finish_op 16 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.695+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 5 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.695+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 16 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.695+0000 7f540c90f640 5 client.4379.objecter 1 in flight 2023-12-15T17:03:36.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 22 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:36.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-15T17:03:36.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:37.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 17 ondisk uv 0 in 7.1 attempt 0 2023-12-15T17:03:37.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540c90f640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-15T17:03:37.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:37.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 17 2023-12-15T17:03:37.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540c90f640 15 client.4379.objecter _finish_op 17 2023-12-15T17:03:37.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:37.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 17 2023-12-15T17:03:37.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:37.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (16 ms) 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 20 librados: queue_aio_write 0x55c5ae96e700 completion 0x55c5aeb551c0 write_seq 4 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 31 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 18 osd.2 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 18 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 15 client.4379.objecter _send_op 18 to 7.1 on osd.2 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:18 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) v8 -- 0x55c5aeb52a70 con 0x55c5aea87cf0 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:36.701+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.259+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:37.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.259+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:37.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.259+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:37.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 23 ==== osd_op_reply(18 exist [writefull 0~5] v31'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:37.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(18 exist [writefull 0~5] v31'5 uv5 ondisk = 0) v8 2023-12-15T17:03:37.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:37.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 18 ondisk uv 5 in 7.1 attempt 0 2023-12-15T17:03:37.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 18 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540c90f640 15 client.4379.objecter _finish_op 18 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 18 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540c90f640 20 librados: complete_aio_write 0x55c5aeb551c0 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (780 ms) 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 20 librados: queue_aio_write 0x55c5ae96e700 completion 0x55c5aeb52dd0 write_seq 5 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 31 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:37.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 19 osd.2 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 19 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 15 client.4379.objecter _send_op 19 to 7.1 on osd.2 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:19 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) v8 -- 0x55c5aeb52a70 con 0x55c5aea87cf0 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.482+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.700+0000 7f5409909640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 24 ==== osd_map(32..32 src has 1..32) v4 ==== 636+0+0 (crc 0 0 0) 0x7f54000196e0 con 0x55c5aea87cf0 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.700+0000 7f5409909640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_map(32..32 src has 1..32) v4 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.700+0000 7f5409909640 3 client.4379.objecter handle_osd_map got epochs [32,32] > 31 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.700+0000 7f5409909640 3 client.4379.objecter handle_osd_map decoding incremental epoch 32 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.700+0000 7f5409909640 10 client.4379.objecter checking op 19 2023-12-15T17:03:37.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.700+0000 7f5409909640 20 client.4379.objecter _calc_target epoch 32 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:38.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.700+0000 7f5409909640 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:38.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.700+0000 7f5409909640 20 client.4379.objecter dump_active .. 0 homeless 2023-12-15T17:03:38.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:37.700+0000 7f5409909640 20 client.4379.objecter 19 7.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 25 ==== osd_op_reply(19 exist [writefull 0~5] v32'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(19 exist [writefull 0~5] v32'6 uv6 ondisk = 0) v8 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 19 ondisk uv 6 in 7.1 attempt 0 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 19 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540c90f640 15 client.4379.objecter _finish_op 19 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 19 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540c90f640 20 librados: complete_aio_write 0x55c5aeb52dd0 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (665 ms) 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-12-15T17:03:38.262 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 20 librados: queue_aio_write 0x55c5ae96e700 completion 0x55c5aeb52ee0 write_seq 6 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 10 client.4379.objecter _op_submit op 0x55c5ae9ccf70 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 20 client.4379.objecter _calc_target epoch 32 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 20 client.4379.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 10 client.4379.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 20 client.4379.objecter _get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 10 client.4379.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 20 osd.2 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 20 client.4379.objecter get_session s=0x55c5ae933140 osd=2 3 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 15 client.4379.objecter _session_op_assign 2 20 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 15 client.4379.objecter _send_op 20 to 7.1 on osd.2 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- osd_op(unknown.0.0:20 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c5aeb551c0 con 0x55c5aea87cf0 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:38.263 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.147+0000 7f540ee3b780 5 client.4379.objecter 1 in flight 2023-12-15T17:03:38.264 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.259+0000 7f5408907640 10 monclient: tick 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.259+0000 7f5408907640 10 monclient: _check_auth_tickets 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.259+0000 7f5408907640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 <== osd.2 v2:172.21.15.69:6800/3322029776 26 ==== osd_op_reply(20 exist [writefull 0~5] v32'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5400014040 con 0x55c5aea87cf0 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 10 client.4379.objecter ms_dispatch 0x55c5ae968460 osd_op_reply(20 exist [writefull 0~5] v32'7 uv7 ondisk = 0) v8 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 10 client.4379.objecter in handle_osd_op_reply 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 7 client.4379.objecter handle_osd_op_reply 20 ondisk uv 7 in 7.1 attempt 0 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 10 client.4379.objecter op 0 rval 0 len 0 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 15 client.4379.objecter handle_osd_op_reply completed tid 20 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 15 client.4379.objecter _finish_op 20 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 4 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 15 client.4379.objecter _session_op_remove 2 20 2023-12-15T17:03:38.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 5 client.4379.objecter 0 in flight 2023-12-15T17:03:38.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 20 librados: complete_aio_write 0x55c5aeb52ee0 2023-12-15T17:03:38.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (410 ms) 2023-12-15T17:03:38.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 10 librados: watch_flush enter 2023-12-15T17:03:38.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 10 librados: watch_flush exit 2023-12-15T17:03:38.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 10 client.4379.objecter close_session for osd.2 2023-12-15T17:03:38.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x55c5aea87cf0 msgr2=0x55c5aea72860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:03:38.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x55c5aea87cf0 0x55c5aea72860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:03:38.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 20 client.4379.objecter put_session s=0x55c5ae933140 osd=2 1 2023-12-15T17:03:38.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 20 client.4379.objecter shutdown clearing up homeless session... 2023-12-15T17:03:38.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 10 client.4379.objecter successfully canceled tick 2023-12-15T17:03:38.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f53f806adc0 msgr2=0x7f53f806d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:03:38.561 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f53f806adc0 0x7f53f806d280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f53fc059040 tx=0x7f53fc005ac0 comp rx=0 tx=0).stop 2023-12-15T17:03:38.561 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 10 monclient: shutdown 2023-12-15T17:03:38.561 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:03:38.562 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c5aea898e0 msgr2=0x55c5ae9d46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 1 --2- 172.21.15.69:0/642570505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c5aea898e0 0x55c5ae9d46d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5404008090 tx=0x7f5404008550 comp rx=0 tx=0).stop 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540c90f640 1 -- 172.21.15.69:0/642570505 reap_dead start 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 shutdown_connections 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 >> 172.21.15.69:0/642570505 conn(0x55c5ae9b9ee0 msgr2=0x55c5ae9f8830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 shutdown_connections 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 1 -- 172.21.15.69:0/642570505 wait complete. 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.557+0000 7f540ee3b780 1 librados: shutdown 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] 12 tests from AsioRados (9019 ms total) 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] Global test environment tear-down 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (17332 ms total) 2023-12-15T17:03:38.563 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ PASSED ] 12 tests. 2023-12-15T17:03:41.621 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:51.566 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:41.625 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:51.571 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:41.635 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:51.575 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:41.638 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:51.577 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:41.674 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:51.619 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:41.676 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:51.621 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:41.873 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:51.812 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:41.874 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:51.819 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:41.953 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:51.898 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout:nown 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 53.802% pgs not active 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 415 creating+peering 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 389 unknown 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 59 creating+activating 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 18 active+clean 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: io: 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: client: 238 B/s wr, 0 op/s rd, 0 op/s wr 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:52.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5428 ms) 2023-12-15T17:03:52.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.Status 2023-12-15T17:03:52.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.Status (20069 ms) 2023-12-15T17:03:52.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] 4 tests from LibRadosService (31033 ms total) 2023-12-15T17:03:52.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-12-15T17:03:52.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] Global test environment tear-down 2023-12-15T17:03:52.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31033 ms total) 2023-12-15T17:03:52.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ PASSED ] 4 tests. 2023-12-15T17:03:52.742 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] Global test environment set-up. 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5319 ms) 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (310 ms) 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20475 ms) 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: attempt 0 of 20 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5219 ms) 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (31323 ms total) 2023-12-15T17:03:52.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: 2023-12-15T17:03:52.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] Global test environment tear-down 2023-12-15T17:03:52.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (31323 ms total) 2023-12-15T17:03:52.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-12-15T17:03:57.927 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:03:57.833+0000 10a53640 -1 osd.4 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12.c0c27492 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:03:58.858 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:03:58.846+0000 10a53640 -1 osd.4 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12.c0c27492 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:03:59.072 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:03:59.065+0000 10a53640 -1 osd.2 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:00.327 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:00.167+0000 10a53640 -1 osd.2 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:00.666 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:00.662+0000 10a53640 -1 osd.4 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12.c0c27492 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:01.218 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:01.195+0000 10a53640 -1 osd.2 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:01.257 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:11.202 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:01.260 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:11.206 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:01.313 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:11.258 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:01.318 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:11.264 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:01.320 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:11.260 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:01.324 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:11.264 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:01.382 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:11.327 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:01.442 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:11.387 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:01.443 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:11.382 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:01.899 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:01.873+0000 10a53640 -1 osd.4 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12.c0c27492 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:02.419 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:02.409+0000 10a53640 -1 osd.2 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:02.951 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:02.948+0000 10a53640 -1 osd.4 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12.c0c27492 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:03.838 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:03.751+0000 10a53640 -1 osd.2 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:03.990 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:03.986+0000 10a53640 -1 osd.4 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12.c0c27492 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:05.046 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:05.037+0000 10a53640 -1 osd.4 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12.c0c27492 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:05.049 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:05.046+0000 10a53640 -1 osd.2 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:06.095 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:06.092+0000 10a53640 -1 osd.4 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12.c0c27492 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:06.470 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:06.445+0000 10a53640 -1 osd.2 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:07.274 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:07.132+0000 10a53640 -1 osd.4 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12.c0c27492 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:07.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-12-15T17:04:07.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-12-15T17:04:07.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-12-15T17:04:07.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-12-15T17:04:07.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (46134 ms) 2023-12-15T17:04:07.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (46134 ms total) 2023-12-15T17:04:07.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: 2023-12-15T17:04:07.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-12-15T17:04:07.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (46134 ms total) 2023-12-15T17:04:07.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-12-15T17:04:07.602 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:07.526+0000 10a53640 -1 osd.2 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:08.749 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:08.568+0000 10a53640 -1 osd.2 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:08.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: Running main() from gmock_main.cc 2023-12-15T17:04:08.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-12-15T17:04:08.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] Global test environment set-up. 2023-12-15T17:04:08.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-12-15T17:04:08.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-12-15T17:04:08.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolList (5774 ms) 2023-12-15T17:04:08.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-12-15T17:04:08.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3947 ms) 2023-12-15T17:04:08.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3812 ms) 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4119 ms) 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3728 ms) 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6204 ms) 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5551 ms) 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5268 ms) 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (9115 ms) 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] 9 tests from LibRadosPools (47518 ms total) 2023-12-15T17:04:08.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: 2023-12-15T17:04:08.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] Global test environment tear-down 2023-12-15T17:04:08.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (47518 ms total) 2023-12-15T17:04:08.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ PASSED ] 9 tests. 2023-12-15T17:04:08.879 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:08.795+0000 10a53640 -1 osd.4 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12.c0c27492 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:09.965 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:09.835+0000 10a53640 -1 osd.2 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:10.200 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:10.097+0000 10a53640 -1 osd.4 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12:492e4303:test-rados-api-smithi069-92185-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:11.117 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:10.979+0000 10a53640 -1 osd.2 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:11.236 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:11.224+0000 10a53640 -1 osd.4 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12:492e4303:test-rados-api-smithi069-92185-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:12.284 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:12.264+0000 10a53640 -1 osd.4 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12:492e4303:test-rados-api-smithi069-92185-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:12.310 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:12.228+0000 10a53640 -1 osd.2 65 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:13.353 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:13.261+0000 10a53640 -1 osd.2 65 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:13.514 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:13.411+0000 10a53640 -1 osd.4 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12:492e4303:test-rados-api-smithi069-92185-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:14.552 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:14.509+0000 10a53640 -1 osd.2 66 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:14.679 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:14.582+0000 10a53640 -1 osd.4 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12:492e4303:test-rados-api-smithi069-92185-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:15.654 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:04:15.583+0000 10a53640 -1 osd.4 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:2 12.12 12:492e4303:test-rados-api-smithi069-92185-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-15T17:04:15.756 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:15.746+0000 10a53640 -1 osd.2 67 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4448.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-12-15T17:04:16.910 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:16.892+0000 10a53640 -1 osd.2 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:3 13.f 13:f4b50b61:::1.obj:head [write 0~127 in=127b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) 2023-12-15T17:04:18.085 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:18.014+0000 10a53640 -1 osd.2 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:3 13.f 13:f4b50b61:::1.obj:head [write 0~127 in=127b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) 2023-12-15T17:04:19.088 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:19.077+0000 10a53640 -1 osd.2 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:3 13.f 13:f4b50b61:::1.obj:head [write 0~127 in=127b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) 2023-12-15T17:04:20.263 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:20.186+0000 10a53640 -1 osd.2 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:3 13.f 13:f4b50b61:::1.obj:head [write 0~127 in=127b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) 2023-12-15T17:04:21.315 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:21.293+0000 10a53640 -1 osd.2 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:3 13.f 13:f4b50b61:::1.obj:head [write 0~127 in=127b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) 2023-12-15T17:04:21.429 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:31.375 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:21.432 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:31.377 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:21.498 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:31.444 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:21.501 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:31.440 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:21.501 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:31.446 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:21.513 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:31.452 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:21.613 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:31.558 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:21.624 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:31.564 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:21.786 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:31.731 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:22.252 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:04:22.112+0000 10a53640 -1 osd.0 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4468.0:7 28.3s0 28:d228a238:test-rados-api-smithi069-92458-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2023-12-15T17:04:22.538 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:04:22.525+0000 10a53640 -1 osd.2 73 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4452.0:3 13.f 13:f4b50b61:::1.obj:head [write 0~127 in=127b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) 2023-12-15T17:04:24.367 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:04:24.225+0000 10a53640 -1 osd.0 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4468.0:7 28.3s0 28:d228a238:test-rados-api-smithi069-92458-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2023-12-15T17:04:25.528 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:04:25.511+0000 10a53640 -1 osd.0 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4468.0:7 28.3s0 28:d228a238:test-rados-api-smithi069-92458-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2023-12-15T17:04:26.725 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:04:26.723+0000 10a53640 -1 osd.0 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4468.0:7 28.3s0 28:d228a238:test-rados-api-smithi069-92458-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2023-12-15T17:04:28.240 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:04:28.227+0000 10a53640 -1 osd.0 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4468.0:7 28.3s0 28:d228a238:test-rados-api-smithi069-92458-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2023-12-15T17:04:29.363 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:04:29.307+0000 10a53640 -1 osd.0 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4468.0:7 28.3s0 28:d228a238:test-rados-api-smithi069-92458-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2023-12-15T17:04:31.599 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:04:31.484+0000 10a53640 -1 osd.0 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4468.0:7 28.3s0 28:d228a238:test-rados-api-smithi069-92458-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2023-12-15T17:04:41.629 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:51.574 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:41.696 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:51.642 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:41.760 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:51.699 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:41.764 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:51.704 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:41.776 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:51.721 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:41.780 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:51.725 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:41.869 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:51.809 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:41.876 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:51.821 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:41.909 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:51.855 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:04:45.432 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2023-12-15T17:04:45.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] Global test environment set-up. 2023-12-15T17:04:45.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-12-15T17:04:45.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-12-15T17:04:45.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-12-15T17:04:45.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-12-15T17:04:45.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-12-15T17:04:45.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-12-15T17:04:45.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-12-15T17:04:45.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-12-15T17:03:21.533+0000 7ffa34236780 -1 failed for service _ceph-mon._tcp 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-12-15T17:03:21.533+0000 7ffa34236780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (283 ms) 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2008 ms) 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2291 ms total) 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x56229784dd40 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562297a68fd0 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562297a698a0 2023-12-15T17:04:45.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5622979b44f0 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5622979b5c10 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562297bdfbd0 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562297b3ae80 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562297be0f10 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562297be1940 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562297be2360 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562297be16b0 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562297be3210 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562297be2100 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5622979d4b50 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562297be2fe0 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5622979d5a30 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5622979d48d0 2023-12-15T17:04:45.435 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5622979d68b0 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x5622979d57e0 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 19 aios 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x56229784dd40 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562297a68fd0 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562297a698a0 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5622979b44f0 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5622979b5c10 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562297bdfbd0 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562297b3ae80 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562297be0f10 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562297be1940 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562297be2360 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562297be16b0 2023-12-15T17:04:45.436 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562297be3210 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562297be2100 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5622979d4b50 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562297be2fe0 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5622979d5a30 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5622979d48d0 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5622979d68b0 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x5622979d57e0 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: done. 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (25264 ms) 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (25264 ms total) 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-12-15T17:04:45.437 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-12-15T17:04:45.438 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-12-15T17:04:45.438 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.Exec (41798 ms) 2023-12-15T17:04:45.438 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-12-15T17:04:48.326 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: Running main() from gmock_main.cc 2023-12-15T17:04:48.326 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-12-15T17:04:48.326 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-12-15T17:04:48.326 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-12-15T17:04:48.326 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: seed 92452 2023-12-15T17:04:48.326 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-12-15T17:04:48.326 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (11625 ms) 2023-12-15T17:04:48.326 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-12-15T17:04:48.326 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (9018 ms) 2023-12-15T17:04:48.326 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-12-15T17:04:48.326 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (49 ms) 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (5850 ms) 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (27428 ms) 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (53970 ms total) 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (6626 ms) 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (13 ms) 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (2139 ms) 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-12-15T17:04:48.327 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (12489 ms) 2023-12-15T17:04:48.328 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (21267 ms total) 2023-12-15T17:04:48.328 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: 2023-12-15T17:04:48.328 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-12-15T17:04:48.328 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (86996 ms total) 2023-12-15T17:04:48.328 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-12-15T17:05:04.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (1050 ms) 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (206 ms) 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (101639 ms) 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (102895 ms total) 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (102895 ms total) 2023-12-15T17:05:04.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-12-15T17:05:07.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: Running main() from gmock_main.cc 2023-12-15T17:05:07.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-12-15T17:05:07.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-12-15T17:05:07.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-12-15T17:05:07.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-12-15T17:05:07.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-12-15T17:05:07.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-12-15T17:05:07.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (23320 ms) 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (6726 ms) 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (7475 ms) 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (20467 ms) 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (5747 ms) 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (7350 ms) 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (6144 ms) 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (6806 ms) 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-12-15T17:05:07.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5852 ms) 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (1537 ms) 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (101 ms) 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (95 ms) 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (1624 ms) 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (789 ms) 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (383 ms) 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (147 ms) 2023-12-15T17:05:07.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (94563 ms total) 2023-12-15T17:05:07.740 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: 2023-12-15T17:05:07.740 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-12-15T17:05:07.740 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (106280 ms total) 2023-12-15T17:05:07.740 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-12-15T17:05:11.776 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: Running main() from gmock_main.cc 2023-12-15T17:05:11.776 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-12-15T17:05:11.776 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] Global test environment set-up. 2023-12-15T17:05:11.776 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-12-15T17:05:11.776 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-12-15T17:05:11.776 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1159 ms) 2023-12-15T17:05:11.776 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-12-15T17:05:11.776 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (273 ms) 2023-12-15T17:05:11.776 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-12-15T17:05:11.776 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (100942 ms) 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:04:59.489331+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-92126-10"}]: dispatch 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:00.320991+0000 mon.b [INF] from='client.? 172.21.15.69:0/2617667492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-92112-10"}]: dispatch 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:00.533197+0000 mon.b [INF] from='client.? 172.21.15.69:0/1702718168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-92126-10"}]: dispatch 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:01.671480+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-92112-10"}]: dispatch 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:01.684165+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-92126-10"}]: dispatch 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:02.418909+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-92319-6"}]': finished 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:02.420768+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-92458-6", "pg_num": 4}]': finished 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:03.175657+0000 mon.a [INF] from='client.? 172.21.15.69:0/2199342414' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-92611-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:03.808089+0000 mon.b [INF] from='client.? 172.21.15.69:0/4262507834' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-92319-6", "tierpool": "test-rados-api-smithi069-92319-7"}]: dispatch 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:03.901386+0000 mon.b [INF] from='client.? 172.21.15.69:0/2617667492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-92112-10"}]: dispatch 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:04.513002+0000 mon.b [INF] from='client.5926 172.21.15.69:0/1702718168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-92126-10"}]: dispatch 2023-12-15T17:05:11.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:05.187272+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-92319-6", "tierpool": "test-rados-api-smithi069-92319-7"}]: dispatch 2023-12-15T17:05:11.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:05.215702+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-92112-10"}]: dispatch 2023-12-15T17:05:11.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:05.246414+0000 mon.a [INF] from='client.5926 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-92126-10"}]: dispatch 2023-12-15T17:05:18.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:04.399483+0000 mon.c [I api_list: [==========] Running 11 tests from 3 test suites. 2023-12-15T17:05:18.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] Global test environment set-up. 2023-12-15T17:05:18.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 7 tests from LibRadosList 2023-12-15T17:05:18.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-12-15T17:05:18.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjects (90097 ms) 2023-12-15T17:05:18.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-12-15T17:05:18.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (4221 ms) 2023-12-15T17:05:18.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-12-15T17:05:18.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo2,foo3 2023-12-15T17:05:18.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-12-15T17:05:18.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo2 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo3 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo4,foo5 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo4 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo5 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo6,foo7 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo7 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo6 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo4 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo5 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo7 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo6 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo1 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo1 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo2 2023-12-15T17:05:18.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo3 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (5661 ms) 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 1 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 10 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 13 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 7 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 14 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 0 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 15 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 11 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 5 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 8 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 6 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 3 0 2023-12-15T17:05:18.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 4 0 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 12 0 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 9 0 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2 0 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (8988 ms) 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: x cursor=MIN 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2023-12-15T17:05:18.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: FIRST> seek to MIN oid=1 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:02547ec2:::1:head 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:02547ec2:::1:head -> 1 2023-12-15T17:05:18.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2023-12-15T17:05:18.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:52ea6a34:::10:head 2023-12-15T17:05:18.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2023-12-15T17:05:18.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2023-12-15T17:05:18.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:566253c9:::13:head 2023-12-15T17:05:18.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:566253c9:::13:head -> 13 2023-12-15T17:05:18.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2023-12-15T17:05:18.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2023-12-15T17:05:18.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2023-12-15T17:05:20.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=14 cursor=8:62a1935d:::14:head 2023-12-15T17:05:20.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:62a1935d:::14:head 2023-12-15T17:05:20.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:62a1935d:::14:head -> 14 2023-12-15T17:05:20.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2023-12-15T17:05:20.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:6cac518f:::0:head 2023-12-15T17:05:20.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:6cac518f:::0:head -> 0 2023-12-15T17:05:20.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2023-12-15T17:05:20.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:863748b0:::15:head 2023-12-15T17:05:20.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:863748b0:::15:head -> 15 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:89d3ae78:::11:head 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:b29083e3:::5:head 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:b29083e3:::5:head -> 5 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2023-12-15T17:05:20.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:cfc208b3:::3:head 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:d83876eb:::4:head 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:d83876eb:::4:head -> 4 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:e960b815:::9:head 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:e960b815:::9:head -> 9 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:f905c69b:::2:head 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:f905c69b:::2:head -> 2 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:52ea6a34:::10:head 2023-12-15T17:05:20.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=10 expected=10 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:f905c69b:::2:head 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:f905c69b:::2:head -> 2 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=2 expected=2 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:e960b815:::9:head 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:e960b815:::9:head -> 9 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=9 expected=9 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=12 expected=12 2023-12-15T17:05:20.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:d83876eb:::4:head 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:d83876eb:::4:head -> 4 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=4 expected=4 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:cfc208b3:::3:head 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=3 expected=3 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=6 expected=6 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:6cac518f:::0:head 2023-12-15T17:05:20.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2023-12-15T17:05:20.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:6cac518f:::0:head -> 0 2023-12-15T17:05:20.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=0 expected=0 2023-12-15T17:05:20.261 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:62a1935d:::14:head 2023-12-15T17:05:21.812 INFO:tasks.workunit.client.0.smithi069.stdout: apNF] from='client.? 172.21.15.69:0/2287329338' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-92458-4", "tierpool": "test-rados-api-smithi069-92458-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-12-15T17:05:21.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:05.642838+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-92458-4", "tierpool": "test-rados-api-smithi069-92458-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-12-15T17:05:21.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:05.660373+0000 mon.a [INF] from='client.? 172.21.15.69:0/2199342414' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-92611-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-15T17:05:21.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:06.004627+0000 mon.b [INF] from='client.? 172.21.15.69:0/1122151210' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-12-15T17:05:21.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:06.960071+0000 client.admin [INF] onexx 2023-12-15T17:05:21.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:06.966943+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-12-15T17:05:21.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:07.047526+0000 mon.b [INF] from='client.? 172.21.15.69:0/2617667492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-92112-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-15T17:05:21.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:07.467169+0000 mon.b [INF] from='client.5926 172.21.15.69:0/1702718168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-92126-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-15T17:05:21.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:10.950075+0000 mon.c [INF] from='client.? 172.21.15.69:0/2287329338' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-92458-4", "overlaypool": "test-rados-api-smithi069-92458-6"}]: dispatch 2023-12-15T17:05:21.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:12.276331+0000 mon.b [INF] from='client.? 172.21.15.69:0/1122151210' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-12-15T17:05:21.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:12.790466+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-92458-4", "overlaypool": "test-rados-api-smithi069-92458-6"}]: dispatch 2023-12-15T17:05:21.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:12.822675+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-92112-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-15T17:05:21.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:12.825381+0000 mon.a [INF] from='client.5926 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-92126-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-15T17:05:21.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:12.828516+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-92392-7"}]': finished 2023-12-15T17:05:21.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:13.703950+0000 mon.a [INF] from='client.8672 172.21.15.69:0/2200878645' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-92056-22"}]: dispatch 2023-12-15T17:05:21.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:13.855662+0000 mon.a [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-15T17:05:22.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:14.169367+0000 client.admin [INF] twoxx 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:14.218213+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:14.463294+0000 mon.a [INF] from='client.? 172.21.15.69:0/2827078895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-92009-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:14.915862+0000 mon.a [INF] from='client.8672 172.21.15.69:0/2200878645' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-92056-22"}]: dispatch 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:14.052093+0000 mon.b [INF] from='client.? 172.21.15.69:0/2617667492' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-92112-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-92112-10"}]: dispatch 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:14.067474+0000 mon.b [INF] from='client.? 172.21.15.69:0/3562450963' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-92047-16"}]: dispatch 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:14.074003+0000 mon.b [INF] from='client.? 172.21.15.69:0/1288167568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-92479-12"}]: dispatch 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:14.323517+0000 mon.c [INF] from='client.? 172.21.15.69:0/3075309808' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-92392-7"}]: dispatch 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:14.367664+0000 mon.b [INF] from='client.5926 172.21.15.69:0/1702718168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-92126-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-92126-10"}]: dispatch 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:14.889769+0000 mon.c [INF] from='client.? 172.21.15.69:0/1262043082' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-92026-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:16.112792+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-92112-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-92112-10"}]: dispatch 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:16.118420+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-92047-16"}]: dispatch 2023-12-15T17:05:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:16.123300+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-92479-12"}]: dispatch 2023-12-15T17:05:22.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:16.132279+0000 mon.a [INF] from='client.5926 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-92126-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-92126-10"}]: dispatch 2023-12-15T17:05:22.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:16.484678+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-92392-7"}]: dispatch 2023-12-15T17:05:22.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:16.515072+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-92458-4", "overlaypool": "test-rados-api-smithi069-92458-6"}]': finished 2023-12-15T17:05:24.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:16.778068+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"pref api_stat: Running main() from gmock_main.cc 2023-12-15T17:05:24.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-12-15T17:05:24.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] Global test environment set-up. 2023-12-15T17:05:24.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-12-15T17:05:24.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-12-15T17:05:24.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.Stat (22101 ms) 2023-12-15T17:05:24.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-12-15T17:05:24.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (11464 ms) 2023-12-15T17:05:24.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-12-15T17:05:24.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.StatNS (30817 ms) 2023-12-15T17:05:24.240 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (4 ms) 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2752 ms) 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 5 tests from LibRadosStat (67138 ms total) 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (10992 ms) 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (9323 ms) 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1718 ms) 2023-12-15T17:05:24.241 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-12-15T17:05:24.242 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1500 ms) 2023-12-15T17:05:24.242 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (23533 ms total) 2023-12-15T17:05:24.242 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: 2023-12-15T17:05:24.242 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] Global test environment tear-down 2023-12-15T17:05:24.242 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (122922 ms total) 2023-12-15T17:05:24.242 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ PASSED ] 9 tests. 2023-12-15T17:05:27.308 INFO:tasks.workunit.client.0.smithi069.stdout:ix": "osd pool application enable","pool": "IsCompletePP_smithi069-92026-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-15T17:05:27.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:16.919328+0000 mon.a [INF] from='client.8672 172.21.15.69:0/2200878645' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-92056-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-15T17:05:27.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:17.516793+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on front (longest 9449.567ms) (OSD_SLOW_PING_TIME_FRONT) 2023-12-15T17:05:27.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:17.721858+0000 mon.b [INF] from='client.? 172.21.15.69:0/3562450963' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-92047-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:17.728902+0000 mon.b [INF] from='client.? 172.21.15.69:0/1288167568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-92479-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:18.316108+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:19.058935+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-92047-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:19.083018+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-92479-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:18.352143+0000 mon.c [INF] from='client.? 172.21.15.69:0/2287329338' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-92458-6", "mode": "writeback"}]: dispatch 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:20.215996+0000 mon.a [INF] from='client.? 172.21.15.69:0/2827078895' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-92009-9","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:20.220758+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-92392-7"}]': finished 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:20.223397+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-92026-11","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:20.226056+0000 mon.a [INF] from='client.8672 172.21.15.69:0/2200878645' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-92056-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:20.553750+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-92458-6", "mode": "writeback"}]: dispatch 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:20.765919+0000 mon.a [INF] from='client.8672 172.21.15.69:0/2200878645' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-92056-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-92056-22"}]: dispatch 2023-12-15T17:05:27.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:21.225108+0000 mon.a [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-15T17:05:33.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-12-15T17:05:33.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-12-15T17:05:33.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-12-15T17:05:33.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-12-15T17:05:33.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-12-15T17:05:33.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2023-12-15T17:05:33.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (23933 ms) 2023-12-15T17:05:33.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-12-15T17:05:33.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (47873 ms) 2023-12-15T17:05:33.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (71806 ms total) 2023-12-15T17:05:33.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: 2023-12-15T17:05:33.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (13252 ms) 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (30471 ms) 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (265 ms) 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (200 ms) 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94684051462656 notify_id 489626271747 notifier_gid 4655 2023-12-15T17:05:33.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (1129 ms) 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94684052548368 notify_id 493921239040 notifier_gid 4655 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (792 ms) 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94684052548368 notify_id 493921239040 notifier_gid 4655 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (557 ms) 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94684053225840 notify_id 493921239044 notifier_gid 4655 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (402 ms) 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94684053867840 notify_id 493921239043 notifier_gid 4655 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (515 ms) 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-12-15T17:05:33.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94684053867840 notify_id 493921239041 notifier_gid 4655 2023-12-15T17:05:33.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (283 ms) 2023-12-15T17:05:33.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-12-15T17:05:33.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: trying... 2023-12-15T17:05:33.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94684053867840 notify_id 493921239044 notifier_gid 4655 2023-12-15T17:05:38.890 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: timed ou list_parallel: process_1_[92790]: starting. 2023-12-15T17:05:38.890 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92790]: creating pool ceph_test_rados_list_parallel.smithi069-92761 2023-12-15T17:05:38.890 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92790]: created object 0... 2023-12-15T17:05:38.890 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92790]: created object 25... 2023-12-15T17:05:38.890 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92790]: created object 49... 2023-12-15T17:05:38.890 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92790]: finishing. 2023-12-15T17:05:38.890 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[92790]: shutting down. 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92792]: starting. 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92792]: listing objects. 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92792]: listed object 0... 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92792]: listed object 25... 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92792]: saw 50 objects 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[92792]: shutting down. 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[106652]: starting. 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[106652]: creating pool ceph_test_rados_list_parallel.smithi069-92761 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[106652]: created object 0... 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[106652]: created object 25... 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[106652]: created object 49... 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[106652]: finishing. 2023-12-15T17:05:38.891 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[106652]: shutting down. 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[106653]: starting. 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[106653]: listing objects. 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[106653]: listed object 0... 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[106653]: listed object 25... 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[106653]: saw 45 objects 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[106653]: shutting down. 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[106654]: starting. 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[106654]: removed 25 objects... 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[106654]: removed half of the objects 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[106654]: removed 50 objects... 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[106654]: removed 50 objects 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[106654]: shutting down. 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:38.892 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[118224]: starting. 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[118224]: creating pool ceph_test_rados_list_parallel.smithi069-92761 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[118224]: created object 0... 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[118224]: created object 25... 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[118224]: created object 49... 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[118224]: finishing. 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[118224]: shutting down. 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[118225]: starting. 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[118225]: listing objects. 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[118225]: listed object 0... 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[118225]: listed object 25... 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[118225]: listed object 50... 2023-12-15T17:05:38.893 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[118225]: saw 55 objects 2023-12-15T17:05:38.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[118225]: shutting down. 2023-12-15T17:05:38.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:38.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:38.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[118226]: starting. 2023-12-15T17:05:38.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[118226]: added 25 objects... 2023-12-15T17:05:38.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[118226]: added half of the objects 2023-12-15T17:05:38.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[118226]: added 50 objects... 2023-12-15T17:05:38.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[118226]: added 50 objects 2023-12-15T17:05:38.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[118226]: shutting down. 2023-12-15T17:05:38.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:38.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:40.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:21.292508+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:21.646456+0000 mon.b [INF] from='client.? 172.21.15.69:0/1122151210' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:23.388879+0000 client.admin [INF] threexx 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:23.395689+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:23.909021+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-92112-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-92112-10"}]': finished 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:23.927596+0000 mon.a [INF] from='client.5926 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-92126-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-92126-10"}]': finished 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:23.929946+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-92047-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:23.932526+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-92479-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:23.934655+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-92458-6", "mode": "writeback"}]': finished 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-12-15T17:05:24.912354+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 8398.660ms) (OSD_SLOW_PING_TIME_BACK) 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (36557 ms) 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (138931 ms total) 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: 2023-12-15T17:05:40.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] Global test environment tear-down 2023-12-15T17:05:40.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (138931 ms total) 2023-12-15T17:05:40.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ PASSED ] 4 tests. 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92831]: starting. 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92831]: creating pool ceph_test_rados_delete_pools_parallel.smithi069-92812 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92831]: created object 0... 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92831]: created object 25... 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92831]: created object 49... 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92831]: finishing. 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[92831]: shutting down. 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[92832]: starting. 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[92832]: deleting pool ceph_test_rados_delete_pools_parallel.smithi069-92812 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[92832]: shutting down. 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-12-15T17:05:41.125 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[110943]: starting. 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[110943]: creating pool ceph_test_rados_delete_pools_parallel.smithi069-92812 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[110943]: created object 0... 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[110943]: created object 25... 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[110943]: created object 49... 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[110943]: finishing. 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[110943]: shutting down. 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[110946]: starting. 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[110946]: listing objects. 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[110946]: listed object 0... 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[110946]: listed object 25... 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[110946]: saw 50 objects 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[110946]: shutting down. 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-12-15T17:05:41.126 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[110944]: starting. 2023-12-15T17:05:41.127 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[110944]: deleting pool ceph_test_rados_delete_pools_parallel.smithi069-92812 2023-12-15T17:05:41.127 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[110944]: shutting down. 2023-12-15T17:05:41.127 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-12-15T17:05:41.127 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-12-15T17:05:41.127 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-12-15T17:05:43.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: Running main() from gmock_main.cc 2023-12-15T17:05:43.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-12-15T17:05:43.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] Global test environment set-up. 2023-12-15T17:05:43.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-12-15T17:05:43.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: seed 92056 2023-12-15T17:05:43.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-12-15T17:05:43.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-12-15T17:05:43.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (16538 ms) 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (3553 ms) 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (33593 ms) 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (1293 ms) 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (1116 ms) 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2995 ms) 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (3178 ms) 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-12-15T17:05:43.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (8940 ms) 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (7936 ms) 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (3934 ms) 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (2790 ms) 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (6658 ms) 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (404 ms) 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (139 ms) 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-12-15T17:05:43.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (2249 ms) 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (942 ms) 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (899 ms) 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (129 ms) 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (136 ms) 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (97423 ms total) 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (12751 ms) 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (609 ms) 2023-12-15T17:05:43.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-12-15T17:05:43.047 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (307 ms) 2023-12-15T17:05:43.047 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-12-15T17:05:43.047 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (156 ms) 2023-12-15T17:05:43.047 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-12-15T17:05:44.032 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.Ro open_pools_parallel: process_1_[92801]: starting. 2023-12-15T17:05:44.032 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92801]: creating pool ceph_test_rados_open_pools_parallel.smithi069-92787 2023-12-15T17:05:44.032 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92801]: created object 0... 2023-12-15T17:05:44.032 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92801]: created object 25... 2023-12-15T17:05:44.032 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92801]: created object 49... 2023-12-15T17:05:44.032 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92801]: finishing. 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[92801]: shutting down. 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[92802]: starting. 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[92802]: rados_pool_create. 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[92802]: rados_ioctx_create. 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[92802]: shutting down. 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[110866]: starting. 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[110866]: creating pool ceph_test_rados_open_pools_parallel.smithi069-92787 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[110866]: created object 0... 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[110866]: created object 25... 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[110866]: created object 49... 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[110866]: finishing. 2023-12-15T17:05:44.033 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[110866]: shutting down. 2023-12-15T17:05:44.034 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-12-15T17:05:44.034 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[110868]: starting. 2023-12-15T17:05:44.034 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[110868]: rados_pool_create. 2023-12-15T17:05:44.034 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[110868]: rados_ioctx_create. 2023-12-15T17:05:44.034 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[110868]: shutting down. 2023-12-15T17:05:44.034 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-12-15T17:05:44.034 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-12-15T17:05:44.034 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-12-15T17:05:56.261 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[130693]: starting. 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[130693]: creating pool ceph_test_rados_list_parallel.smithi069-92761 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[130693]: created object 0... 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[130693]: created object 25... 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[130693]: created object 49... 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[130693]: finishing. 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[130693]: shutting down. 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[130695]: starting. 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[130695]: listing objects. 2023-12-15T17:05:56.262 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[130695]: listed object 0... 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[130695]: listed object 25... 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[130695]: listed object 50... 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[130695]: listed object 75... 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[130695]: listed object 100... 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[130695]: saw 103 objects 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[130695]: shutting down. 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[130696]: starting. 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[130696]: added 25 objects... 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[130696]: added half of the objects 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[130696]: added 50 objects... 2023-12-15T17:05:56.263 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[130696]: added 50 objects 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[130696]: shutting down. 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[130699]: starting. 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[130699]: removed 25 objects... 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[130699]: removed half of the objects 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[130699]: removed 50 objects... 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[130699]: removed 50 objects 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[130699]: shutting down. 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[130698]: starting. 2023-12-15T17:05:56.264 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[130698]: added 25 objects... 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[130698]: added half of the objects 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[130698]: added 50 objects... 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[130698]: added 50 objects 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[130698]: shutting down. 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[144870]: starting. 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[144870]: creating pool ceph_test_rados_list_parallel.smithi069-92761 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[144870]: created object 0... 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[144870]: created object 25... 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[144870]: created object 49... 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[144870]: finishing. 2023-12-15T17:05:56.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[144870]: shutting down. 2023-12-15T17:05:56.266 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.266 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.266 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.266 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:05:56.266 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[144871]: starting. 2023-12-15T17:06:01.353 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[144 api_lock: Running main() from gmock_main.cc 2023-12-15T17:06:01.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] Global test environment set-up. 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (46681 ms) 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockShared (1392 ms) 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (9231 ms) 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (5962 ms) 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (5711 ms) 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.Unlock (5325 ms) 2023-12-15T17:06:01.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (4819 ms) 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (7787 ms) 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLock (86908 ms total) 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (6718 ms) 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (152 ms) 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (2045 ms) 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1883 ms) 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-12-15T17:06:01.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (195 ms) 2023-12-15T17:06:01.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-12-15T17:06:01.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (220 ms) 2023-12-15T17:06:01.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-12-15T17:06:01.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (296 ms) 2023-12-15T17:06:01.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-12-15T17:06:01.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (207 ms) 2023-12-15T17:06:01.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (11716 ms total) 2023-12-15T17:06:01.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: 2023-12-15T17:06:01.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] Global test environment tear-down 2023-12-15T17:06:01.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (160133 ms total) 2023-12-15T17:06:01.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ PASSED ] 16 tests. 2023-12-15T17:06:01.372 INFO:tasks.workunit.client.0.smithi069.stdout:871]: listing objects. 2023-12-15T17:06:01.372 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[144871]: listed object 0... 2023-12-15T17:06:01.372 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[144871]: listed object 25... 2023-12-15T17:06:01.372 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[144871]: listed object 50... 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[144871]: listed object 75... 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[144871]: listed object 100... 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[144871]: listed object 125... 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[144871]: saw 150 objects 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[144871]: shutting down. 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[144872]: starting. 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[144872]: added 25 objects... 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[144872]: added half of the objects 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[144872]: added 50 objects... 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[144872]: added 50 objects 2023-12-15T17:06:01.373 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[144872]: shutting down. 2023-12-15T17:06:01.374 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:06:01.374 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:06:01.374 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:06:01.374 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:06:01.374 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-12-15T17:06:01.374 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******* SUCCESS ********** 2023-12-15T17:06:01.390 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (12646 ms) 2023-12-15T17:06:01.390 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-12-15T17:06:01.390 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (361 ms) 2023-12-15T17:06:01.390 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-12-15T17:06:01.390 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5776 ms) 2023-12-15T17:06:01.390 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-12-15T17:06:01.390 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-12-15T17:06:01.391 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-12-15T17:06:01.391 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-12-15T17:06:01.391 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-12-15T17:06:01.391 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (68931 ms) 2023-12-15T17:06:01.391 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 8 tests from LibRadosMisc (129513 ms total) 2023-12-15T17:06:01.391 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-12-15T17:06:01.391 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] Global test environment tear-down 2023-12-15T17:06:01.391 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (160129 ms total) 2023-12-15T17:06:01.391 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ PASSED ] 12 tests. 2023-12-15T17:06:02.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: Running main() from gmock_main.cc 2023-12-15T17:06:02.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-12-15T17:06:02.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-12-15T17:06:02.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-12-15T17:06:02.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: seed 92112 2023-12-15T17:06:02.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-12-15T17:06:02.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (15384 ms) 2023-12-15T17:06:02.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-12-15T17:06:02.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (16639 ms) 2023-12-15T17:06:02.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-12-15T17:06:02.511 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (35210 ms) 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (6652 ms) 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (1183 ms) 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (3848 ms) 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (2063 ms) 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (6539 ms) 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (87519 ms total) 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-12-15T17:06:02.512 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (7928 ms) 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (125 ms) 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1662 ms) 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1573 ms) 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (537 ms) 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (571 ms) 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (1261 ms) 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-12-15T17:06:02.513 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (570 ms) 2023-12-15T17:06:02.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (14227 ms total) 2023-12-15T17:06:02.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: 2023-12-15T17:06:02.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-12-15T17:06:02.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (161313 ms total) 2023-12-15T17:06:02.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-12-15T17:06:02.514 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: Running main() from gmock_main.cc 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] Global test environment set-up. 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (28729 ms) 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (33152 ms) 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (8092 ms) 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.Checksum (1562 ms) 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-12-15T17:06:02.515 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (4800 ms) 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (4008 ms) 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (6884 ms) 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (987 ms) 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.TruncTest (1696 ms) 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (3030 ms) 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (161 ms) 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-12-15T17:06:02.516 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RmXattr (1679 ms) 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.XattrIter (232 ms) 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 14 tests from LibRadosIo (95012 ms total) 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (7286 ms) 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (551 ms) 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (393 ms) 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (239 ms) 2023-12-15T17:06:02.517 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (1546 ms) 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (358 ms) 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (181 ms) 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (160 ms) 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (416 ms) 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (334 ms) 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 10 tests from LibRadosIoEC (11465 ms total) 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] Global test environment tear-down 2023-12-15T17:06:02.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [==========] 24 tests from 2 test suites ran. (161352 ms total) 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ PASSED ] 24 tests. 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify_test_cb 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (34182 ms) 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94767077992960 err -107 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (11358 ms) 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-12-15T17:06:02.519 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94767077992960 err -107 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (12399 ms) 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4498 notify_id 330712481792 cookie 94767080401760 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (4258 ms) 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4498 notify_id 352187318272 cookie 94767080401760 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (5582 ms) 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4498 notify_id 369367187457 cookie 94767080457920 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (3432 ms) 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4498 notify_id 399431958530 cookie 94767080457920 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4498 notify_id 399431958530 cookie 94767080508784 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (4676 ms) 2023-12-15T17:06:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4498 notify_id 399431958528 cookie 94767080508784 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4498 notify_id 408021893121 cookie 94767080508784 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (6954 ms) 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94767080508784 err -107 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4498 notify_id 450971566082 cookie 94767080508784 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (8653 ms) 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94767080508784 err -107 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (2062 ms) 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (93556 ms total) 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: 2023-12-15T17:06:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-12-15T17:06:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-12-15T17:06:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify_test_cb 2023-12-15T17:06:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (7605 ms) 2023-12-15T17:06:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (7605 ms total) 2023-12-15T17:06:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: 2023-12-15T17:06:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-12-15T17:06:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (161161 ms total) 2023-12-15T17:06:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-12-15T17:06:02.706 INFO:tasks.workunit.client.0.smithi069.stdout:undTripPP2 (249 ms) 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (255 ms) 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (230 ms) 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (1138 ms) 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (1287 ms) 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (360 ms) 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (191 ms) 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-12-15T17:06:02.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (213 ms) 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (1751 ms) 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (255 ms) 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (207 ms) 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (220 ms) 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (238 ms) 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (20419 ms total) 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] Global test environment tear-down 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (161543 ms total) 2023-12-15T17:06:02.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-12-15T17:06:03.788 INFO:tasks.workunit.client.0.smithi069.stdout:t 2023-12-15T17:06:03.788 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2023-12-15T17:06:03.788 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushed 2023-12-15T17:06:03.788 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3146 ms) 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: trying... 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94684053885664 notify_id 498216206341 notifier_gid 4655 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: timed out 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushed 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3368 ms) 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: List watches 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94684053886704 notify_id 502511173638 notifier_gid 4655 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 done 2023-12-15T17:06:03.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: watch_check 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: unwatch2 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: done 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3471 ms) 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: List watches 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94684053886704 notify_id 506806140929 notifier_gid 4655 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 done 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: watch_check 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: unwatch2 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: done 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3269 ms) 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (61121 ms total) 2023-12-15T17:06:03.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: 2023-12-15T17:06:03.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-12-15T17:06:03.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (162446 ms total) 2023-12-15T17:06:03.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-12-15T17:06:03.802 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:13.747 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:03.819 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:13.764 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:03.884 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:13.823 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:03.885 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:13.825 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:04.017 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:13.956 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:04.055 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:14.001 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:04.066 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:14.012 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:04.079 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:14.025 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:04.260 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:14.205 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:23.062 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: Running main() from gmock_main.cc 2023-12-15T17:06:23.062 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] Global test environment set-up. 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (13740 ms) 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (8362 ms) 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (5610 ms) 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (3509 ms) 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (31221 ms total) 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-12-15T17:06:23.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (18140 ms) 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (8660 ms) 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (18492 ms) 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (45292 ms total) 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (21877 ms) 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (19034 ms) 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (24378 ms) 2023-12-15T17:06:23.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (7792 ms) 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (73081 ms total) 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (5263 ms) 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (6734 ms) 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (11997 ms total) 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] Global test environment tear-down 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (181758 ms total) 2023-12-15T17:06:23.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-12-15T17:06:23.744 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:33.689 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:23.812 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:33.752 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:23.813 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:33.759 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:23.823 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:33.762 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:23.875 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:33.820 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:23.877 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:33.822 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:23.886 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:33.826 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:23.951 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:33.896 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:23.959 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:33.905 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:43.784 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:53.729 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:43.791 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:53.730 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:43.836 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:53.781 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:43.873 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:53.813 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:43.912 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:53.858 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:43.919 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:53.864 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:43.931 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:53.877 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:43.960 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:53.899 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:44.008 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:53.953 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: Running main() from gmock_main.cc 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (37246 ms) 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-12-15T17:06:46.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (12818 ms) 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (11800 ms) 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (16625 ms) 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (7186 ms) 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (7932 ms) 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (111867 ms) 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (205474 ms total) 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (205474 ms total) 2023-12-15T17:06:46.882 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-12-15T17:06:55.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-12-15T17:06:55.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-12-15T17:06:55.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-12-15T17:06:55.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-12-15T17:06:55.928 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: seed 92470 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (28749 ms) 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (21631 ms) 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (6222 ms) 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (5570 ms) 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (16823 ms) 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (78995 ms total) 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-12-15T17:06:55.929 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (19111 ms) 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (33446 ms) 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (27659 ms) 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4232 ms) 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (3655 ms) 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-157-g021ac167/rpm/el9/BUILD/ceph-19.0.0-157-g021ac167/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-12-15T17:06:55.930 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi069-92470-7 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: waiting for snaps to purge 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18760 ms) 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (106863 ms total) 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2892 ms) 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (1957 ms) 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (3173 ms) 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-12-15T17:06:55.931 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2857 ms) 2023-12-15T17:06:55.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (10879 ms total) 2023-12-15T17:06:55.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-12-15T17:06:55.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-12-15T17:07:01.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN api_aio: Running main() from gmock_main.cc 2023-12-15T17:07:01.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-12-15T17:07:01.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] Global test environment set-up. 2023-12-15T17:07:01.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-12-15T17:07:01.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-12-15T17:07:01.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.TooBig (5884 ms) 2023-12-15T17:07:01.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-12-15T17:07:01.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (36055 ms) 2023-12-15T17:07:01.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-12-15T17:07:01.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (9458 ms) 2023-12-15T17:07:01.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (9664 ms) 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (6583 ms) 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (6171 ms) 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (13354 ms) 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (15662 ms) 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (25003 ms) 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (32386 ms) 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-12-15T17:07:01.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (5489 ms) 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3700 ms) 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (4643 ms) 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3345 ms) 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.Flush (3482 ms) 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2659 ms) 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3979 ms) 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4212 ms) 2023-12-15T17:07:01.419 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2939 ms) 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3927 ms) 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3171 ms) 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (4031 ms) 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3893 ms) 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (4145 ms) 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-12-15T17:07:01.420 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3160 ms) 2023-12-15T17:07:01.421 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-12-15T17:07:01.421 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3272 ms) 2023-12-15T17:07:01.421 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 26 tests from LibRadosAio (220267 ms total) 2023-12-15T17:07:01.421 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: 2023-12-15T17:07:01.421 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-12-15T17:07:02.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleW api_aio_pp: Running main() from gmock_main.cc 2023-12-15T17:07:02.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-12-15T17:07:02.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-12-15T17:07:02.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-12-15T17:07:02.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-12-15T17:07:02.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4425 ms) 2023-12-15T17:07:02.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-12-15T17:07:02.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (40910 ms) 2023-12-15T17:07:02.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-12-15T17:07:02.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (11369 ms) 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (8514 ms) 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (7704 ms) 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (9347 ms) 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (7016 ms) 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (14271 ms) 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (29095 ms) 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (27550 ms) 2023-12-15T17:07:02.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (4483 ms) 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3686 ms) 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3246 ms) 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (4651 ms) 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3605 ms) 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3817 ms) 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3827 ms) 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3204 ms) 2023-12-15T17:07:02.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2831 ms) 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3386 ms) 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2746 ms) 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (4022 ms) 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (4007 ms) 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (4052 ms) 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2987 ms) 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-12-15T17:07:02.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3444 ms) 2023-12-15T17:07:02.798 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-12-15T17:07:09.040 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:18.986 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:09.044 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:18.989 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:09.055 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:19.001 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:09.102 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:19.047 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:09.110 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:19.050 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:09.117 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:19.057 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:09.152 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:19.097 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:09.179 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:19.124 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:09.192 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:19.131 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:13.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRado] LibRadosSnapshotsSelfManagedECPP.SnapPP 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4972 ms) 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4078 ms) 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (5050 ms) 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (14100 ms total) 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (231807 ms total) 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-12-15T17:07:13.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-12-15T17:07:28.987 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:38.932 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:28.992 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:38.937 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:29.042 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:38.981 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:29.042 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:38.982 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:29.051 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:38.996 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:29.055 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:39.000 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:29.056 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:39.001 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:29.086 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:39.025 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:29.102 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:39.048 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:34.751 INFO:tasks.workunit.client.0.smithi069.stdout:i_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2023-12-15T17:07:34.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:62a1935d:::14:head -> 14 2023-12-15T17:07:34.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=14 expected=14 2023-12-15T17:07:34.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2023-12-15T17:07:34.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2023-12-15T17:07:34.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2023-12-15T17:07:34.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=7 expected=7 2023-12-15T17:07:34.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:566253c9:::13:head 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:566253c9:::13:head -> 13 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=13 expected=13 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=8 expected=8 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:b29083e3:::5:head 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:b29083e3:::5:head -> 5 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=5 expected=5 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:89d3ae78:::11:head 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2023-12-15T17:07:34.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=11 expected=11 2023-12-15T17:07:34.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:863748b0:::15:head 2023-12-15T17:07:34.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2023-12-15T17:07:34.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:863748b0:::15:head -> 15 2023-12-15T17:07:34.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=15 expected=15 2023-12-15T17:07:34.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 8:02547ec2:::1:head 2023-12-15T17:07:34.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2023-12-15T17:07:34.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 8:02547ec2:::1:head -> 1 2023-12-15T17:07:34.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=1 expected=1 2023-12-15T17:07:34.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (11691 ms) 2023-12-15T17:07:34.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-12-15T17:07:34.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (45333 ms) 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (72015 ms) 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 7 tests from LibRadosList (238006 ms total) 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (2961 ms) 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-12-15T17:07:34.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo2,foo3 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo2 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo3 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo4,foo5 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo4 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo5 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo6,foo7 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo7 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo6 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo4 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo5 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo7 2023-12-15T17:07:34.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo6 2023-12-15T17:07:34.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo1 2023-12-15T17:07:34.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo1 2023-12-15T17:07:34.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo2 2023-12-15T17:07:34.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo3 2023-12-15T17:07:34.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (1268 ms) 2023-12-15T17:07:34.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-12-15T17:07:34.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 1 0 2023-12-15T17:07:34.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 10 0 2023-12-15T17:07:34.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 13 0 2023-12-15T17:07:41.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 7 0 2023-12-15T17:07:41.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 14 0 2023-12-15T17:07:41.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 0 0 2023-12-15T17:07:41.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 15 0 2023-12-15T17:07:41.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 11 0 2023-12-15T17:07:41.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 5 0 2023-12-15T17:07:41.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 8 0 2023-12-15T17:07:41.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 6 0 2023-12-15T17:07:41.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 3 0 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 4 0 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 12 0 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 9 0 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2 0 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (3772 ms) 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 3 tests from LibRadosListEC (8001 ms total) 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3255 ms) 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 1 test from LibRadosListNP (3255 ms total) 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] Global test environment tear-down 2023-12-15T17:07:41.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [==========] 11 tests from 3 test suites ran. (260473 ms total) 2023-12-15T17:07:41.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ PASSED ] 11 tests. 2023-12-15T17:07:49.187 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:59.132 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:49.191 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:59.136 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:49.236 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:59.181 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:49.241 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:59.180 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:49.243 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:59.184 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:49.243 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:59.183 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:49.245 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:59.191 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:49.287 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:59.232 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:07:49.293 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:59.233 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:09.367 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:19.312 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:09.369 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:19.315 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:09.410 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:19.356 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:09.421 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:19.364 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:09.421 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:19.365 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:09.430 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:19.370 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:09.441 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:19.380 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:09.467 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:19.412 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:09.504 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:19.443 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:19.074 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:29.020 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:19.080 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:29.025 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:19.115 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:29.060 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:19.123 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:29.068 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:19.138 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:29.077 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:19.166 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:29.111 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:19.175 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:29.120 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:19.196 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:29.135 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:19.256 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:29.196 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:29.442 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:39.388 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:29.506 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:39.451 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:29.511 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:39.450 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:29.514 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:39.453 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:29.542 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:39.481 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:29.550 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:39.495 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:29.569 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:39.515 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:29.572 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:39.517 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:29.605 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:39.551 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:37.799 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-12-15T17:08:37.799 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-12-15T17:08:37.799 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-12-15T17:08:37.799 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (1 ms total) 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: seed 92185 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (11 ms) 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (63099 ms) 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (5712 ms) 2023-12-15T17:08:37.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (14263 ms) 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (4858 ms) 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (885 ms) 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (1225 ms) 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (1386 ms) 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (400 ms) 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-12-15T17:08:37.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (1536 ms) 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (504 ms) 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (149 ms) 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (249 ms) 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: osd_max_attr_size = 0 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (125898 ms) 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (8834 ms) 2023-12-15T17:08:37.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: done waiting, doing copies 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: waiting for final deep scrubs... 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: done waiting 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (77074 ms) 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (255 ms) 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (82 ms) 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4858 ms) 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-12-15T17:08:37.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-12-15T17:08:37.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-12-15T17:08:49.632 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:59.577 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:49.640 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:59.586 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:49.701 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:59.647 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:49.702 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:59.641 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:49.709 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:59.648 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:49.724 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:59.669 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:49.726 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:59.672 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:49.770 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:59.710 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:08:49.790 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:59.735 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:01.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [------rite 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7577 ms) 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7427 ms) 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7203 ms) 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7822 ms) 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6953 ms) 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7949 ms) 2023-12-15T17:09:01.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (8129 ms) 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (8059 ms) 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6945 ms) 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7496 ms) 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6706 ms) 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6855 ms) 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-12-15T17:09:01.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7267 ms) 2023-12-15T17:09:01.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-12-15T17:09:01.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (9182 ms) 2023-12-15T17:09:01.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-12-15T17:09:01.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7103 ms) 2023-12-15T17:09:01.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-12-15T17:09:01.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7651 ms) 2023-12-15T17:09:01.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (120325 ms total) 2023-12-15T17:09:01.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: 2023-12-15T17:09:01.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] Global test environment tear-down 2023-12-15T17:09:01.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (340592 ms total) 2023-12-15T17:09:01.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ PASSED ] 42 tests. 2023-12-15T17:09:09.183 INFO:tasks.workunit.client.0.smithi069.stdout:----] 22 tests from LibRadosMiscPP (311279 ms total) 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (5013 ms) 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (5013 ms total) 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (742 ms) 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (267 ms) 2023-12-15T17:09:09.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (1009 ms total) 2023-12-15T17:09:09.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-12-15T17:09:09.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-12-15T17:09:09.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-12-15T17:09:09.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (663 ms) 2023-12-15T17:09:09.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-12-15T17:09:09.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (135 ms) 2023-12-15T17:09:09.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (798 ms total) 2023-12-15T17:09:09.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-12-15T17:09:09.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-12-15T17:09:09.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-12-15T17:09:09.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (423 ms) 2023-12-15T17:09:09.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-12-15T17:09:09.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (48 ms) 2023-12-15T17:09:09.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (471 ms total) 2023-12-15T17:09:09.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-12-15T17:09:09.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-12-15T17:09:09.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-12-15T17:09:09.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1510 ms) 2023-12-15T17:09:09.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1510 ms total) 2023-12-15T17:09:09.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-12-15T17:09:09.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-12-15T17:09:09.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (347922 ms total) 2023-12-15T17:09:09.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-12-15T17:09:09.793 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:19.739 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:09.797 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:19.743 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:09.851 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:19.796 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:09.857 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:19.796 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:09.861 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:19.800 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:09.909 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:19.854 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:09.910 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:19.848 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:09.920 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:19.866 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:09.972 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:19.918 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:29.796 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:39.741 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:29.810 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:39.751 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:29.849 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:39.788 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:29.851 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:39.797 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:29.852 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:39.797 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:29.853 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:39.793 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:29.857 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:39.802 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:29.892 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:39.831 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:29.907 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:39.852 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:42.974 INFO:tasks.workunit.client.0.smithi069.stdout:sAio.RoundTripAppendPP (3446 ms) 2023-12-15T17:09:42.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-12-15T17:09:42.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3188 ms) 2023-12-15T17:09:42.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-12-15T17:09:42.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (5061 ms) 2023-12-15T17:09:42.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-12-15T17:09:42.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (4252 ms) 2023-12-15T17:09:42.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-12-15T17:09:42.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: setting pool EIO 2023-12-15T17:09:42.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: max_success 0, min_failed 1 2023-12-15T17:09:42.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (7227 ms) 2023-12-15T17:09:42.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-12-15T17:09:42.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4412 ms) 2023-12-15T17:09:42.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (245781 ms total) 2023-12-15T17:09:42.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-12-15T17:09:42.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-12-15T17:09:42.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-12-15T17:09:42.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3475 ms) 2023-12-15T17:09:42.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-12-15T17:09:42.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9027 ms) 2023-12-15T17:09:42.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-12-15T17:09:42.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (17068 ms) 2023-12-15T17:09:42.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-12-15T17:09:42.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3778 ms) 2023-12-15T17:09:42.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (33348 ms total) 2023-12-15T17:09:42.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3256 ms) 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3256 ms total) 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14514 ms) 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7754 ms) 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-12-15T17:09:42.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6880 ms) 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6883 ms) 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (4328 ms) 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6951 ms) 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (8818 ms) 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7315 ms) 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-12-15T17:09:42.978 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7135 ms) 2023-12-15T17:09:42.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-12-15T17:09:42.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6612 ms) 2023-12-15T17:09:42.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-12-15T17:09:42.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7130 ms) 2023-12-15T17:09:42.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-12-15T17:09:42.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7096 ms) 2023-12-15T17:09:42.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-12-15T17:09:42.979 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (8018 ms) 2023-12-15T17:09:49.863 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:59.808 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:49.914 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:59.860 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:49.956 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:59.901 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:49.967 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:59.906 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:49.971 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:59.911 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:49.980 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:59.924 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:49.982 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:59.928 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:50.024 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:59.963 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:09:50.032 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:59.977 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:04.212 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:14.157 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:04.223 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:14.169 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:04.229 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:14.168 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:04.230 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:14.170 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:04.239 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:14.185 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:04.240 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:14.186 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:04.328 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:14.267 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:04.331 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:14.277 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:04.351 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:14.297 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:14.866 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:24.811 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:14.912 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:24.857 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:14.941 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:24.886 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:14.961 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:24.905 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:14.963 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:24.909 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:14.968 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:24.908 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:14.971 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:24.911 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:14.990 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:24.935 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:14.996 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:24.936 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:28.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-12-15T17:10:28.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3062 ms) 2023-12-15T17:10:28.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-12-15T17:10:28.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7182 ms) 2023-12-15T17:10:28.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-12-15T17:10:28.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6777 ms) 2023-12-15T17:10:28.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-12-15T17:10:28.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7364 ms) 2023-12-15T17:10:28.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-12-15T17:10:28.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7032 ms) 2023-12-15T17:10:28.864 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-12-15T17:10:28.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6588 ms) 2023-12-15T17:10:28.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-12-15T17:10:28.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7882 ms) 2023-12-15T17:10:28.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (145322 ms total) 2023-12-15T17:10:28.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-12-15T17:10:28.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-12-15T17:10:28.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (427707 ms total) 2023-12-15T17:10:28.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-12-15T17:10:34.991 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:44.936 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:34.994 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:44.940 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:35.039 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:44.979 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:35.041 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:44.980 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:35.046 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:44.992 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:35.063 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:45.009 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:35.065 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:45.011 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:35.068 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:45.007 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:35.109 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:45.055 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:55.071 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:05.016 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:55.073 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:05.018 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:55.103 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:05.048 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:55.121 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:05.061 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:55.123 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:05.063 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:55.124 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:05.070 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:55.125 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:05.071 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:55.151 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:05.090 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:10:55.162 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:05.108 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:15.142 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:25.087 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:15.142 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:25.088 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:15.176 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:25.121 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:15.186 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:25.132 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:15.189 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:25.135 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:15.191 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:25.131 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:15.193 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:25.133 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:15.222 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:25.167 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:15.224 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:25.164 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:35.200 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:45.145 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:35.204 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:45.150 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:35.239 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:45.184 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:35.252 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:45.191 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:35.252 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:45.197 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:35.253 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:45.193 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:35.254 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:45.200 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:35.287 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:45.226 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:35.288 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:45.234 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:41.413 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: seed 92319 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (14838 ms) 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (46142 ms) 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (1143 ms) 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (62123 ms total) 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-12-15T17:11:41.414 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (51147 ms) 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (50694 ms) 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (23103 ms) 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [4] 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [5,4] 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [6,5,4] 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting some heads 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 7 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 6 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 5 2023-12-15T17:11:41.415 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 4 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrubs... 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (139759 ms) 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10237 ms) 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (9933 ms) 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7979 ms) 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (9019 ms) 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (12230 ms) 2023-12-15T17:11:41.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9147 ms) 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10030 ms) 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (40059 ms) 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10096 ms) 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (12012 ms) 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (15092 ms) 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8748 ms) 2023-12-15T17:11:41.417 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-12-15T17:11:41.418 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11202 ms) 2023-12-15T17:11:55.300 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:05.245 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:55.301 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:05.247 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:55.346 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:05.285 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:55.347 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:05.293 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:55.348 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:05.288 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:55.359 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:05.304 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:55.360 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:05.306 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:55.383 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:05.322 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:11:55.401 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:05.346 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:03.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-12-15T17:12:03.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: hmm, no HitSet yet 2023-12-15T17:12:03.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: ok, hit_set contains 200:602f83fe:::foo:head 2023-12-15T17:12:03.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (10058 ms) 2023-12-15T17:12:03.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-12-15T17:12:03.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg_num = 32 2023-12-15T17:12:03.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 0 ls 1702660319,0 2023-12-15T17:12:03.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 1 ls 1702660319,0 2023-12-15T17:12:03.954 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 2 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 3 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 4 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 5 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 6 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 7 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 8 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 9 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 10 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 11 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 12 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 13 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 14 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 15 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 16 ls 1702660319,0 2023-12-15T17:12:03.955 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 17 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 18 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 19 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 20 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 21 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 22 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 23 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 24 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 25 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 26 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 27 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 28 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 29 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 30 ls 1702660319,0 2023-12-15T17:12:03.956 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 31 ls 1702660319,0 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg_num = 32 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6cac518f:::0:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:02547ec2:::1:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f905c69b:::2:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:cfc208b3:::3:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d83876eb:::4:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b29083e3:::5:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c4fdafeb:::6:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:5c6b0b28:::7:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:bd63b0f1:::8:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e960b815:::9:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:52ea6a34:::10:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:89d3ae78:::11:head 2023-12-15T17:12:03.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:de5d7c5f:::12:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:566253c9:::13:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:62a1935d:::14:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:863748b0:::15:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3958e169:::16:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4d4dabf9:::17:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:8391935d:::18:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:28883081:::19:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:69259c59:::20:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4bdb80b7:::21:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a11c5d71:::22:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:271af37b:::23:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:95b121be:::24:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:58d1031b:::25:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:0a050783:::26:head 2023-12-15T17:12:03.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c709704c:::27:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:cbe56eaf:::28:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:86b4b162:::29:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:70d89383:::30:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:dd450c7c:::31:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6d5729b1:::32:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c388f3fb:::33:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:56cfea31:::34:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9dbc1bf7:::35:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:40b74ccd:::36:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4d5aaf42:::37:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:920f362c:::38:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6cc53222:::39:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9cad833f:::40:head 2023-12-15T17:12:03.959 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:1ea84d41:::41:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c4480ef6:::42:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a694361e:::43:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d1bd33e9:::44:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ddc2cd5d:::45:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:2b782207:::46:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7b187fca:::47:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:90ecdf6f:::48:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a5ed95fe:::49:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ea0eaa55:::50:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f33ef17b:::51:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a0d1b2f6:::52:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:60c5229e:::53:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:edcbc575:::54:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:102cf253:::55:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:efb7fb0b:::56:head 2023-12-15T17:12:03.960 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:50d0a326:::57:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d4dc5daf:::58:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3a130462:::59:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ec87ed71:::60:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d5bc9454:::61:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3ddfe313:::62:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7c2816b9:::63:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:47e00e4d:::64:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c6410c18:::65:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b48ed237:::66:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:cd63ad31:::67:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b179e92b:::68:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:0d9f741a:::69:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6d3352ae:::70:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c6d5c19e:::71:head 2023-12-15T17:12:03.961 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:bc4729c3:::72:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:77e930b9:::73:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:0abeecfd:::74:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b7c37e15:::75:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b6378398:::76:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:02bd68de:::77:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:cc795d2d:::78:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:630d4fea:::79:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e0d29ef5:::80:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:fd6f13d2:::81:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:606461d5:::82:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:eadbdc43:::83:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:8761d0bb:::84:head 2023-12-15T17:12:03.962 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9ef0186f:::85:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e0d41294:::86:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:961de695:::87:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:1423148f:::88:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:633a8fa2:::89:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a8653809:::90:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3dac8b33:::91:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:35aad435:::92:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f6dcc343:::93:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:dbbdad87:::94:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:1cb48ce0:::95:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:03cd461c:::96:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:17a4ea99:::97:head 2023-12-15T17:12:03.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9993c9a7:::98:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6394211c:::99:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:94c7ae57:::100:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6fdee5bb:::101:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9a477fd1:::102:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:eb850916:::103:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:affc56b9:::104:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b42dc814:::105:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f319f8f0:::106:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9a40b9de:::107:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:8b524f28:::108:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e3de589f:::109:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:90f90a5b:::110:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a7b4f1d7:::111:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:af51766e:::112:head 2023-12-15T17:12:03.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b6f90bd1:::113:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e0261208:::114:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c9569ef7:::115:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:61bebe50:::116:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:fe93412b:::117:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d3d38bee:::118:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3100ba0c:::119:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d0560ada:::120:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f0ea8b35:::121:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:766f231a:::122:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a07a2582:::123:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:bd7c6b3a:::124:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:fb2ddaff:::125:head 2023-12-15T17:12:03.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4408e1fe:::126:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ee1df7a7:::127:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c3002909:::128:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4f48ffa9:::129:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:edf38733:::130:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c08425c0:::131:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:5f902d98:::132:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:41ea2c93:::133:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:813cee13:::134:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:0131818d:::135:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:26ba5a85:::136:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:381b8a5a:::137:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:28797e47:::138:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:bfca7f22:::139:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:36807075:::140:head 2023-12-15T17:12:03.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:80b03975:::141:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:5c15709b:::142:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f39ea15e:::143:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ea992956:::144:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:48887b1c:::145:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:9f24a9dd:::146:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:987f100b:::147:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d2dd3581:::148:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7fed1808:::149:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c80b70e9:::150:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:85ed90f9:::151:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:36428b24:::152:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d044c34a:::153:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7c18bf58:::154:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d1c21232:::155:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a7a3c575:::156:head 2023-12-15T17:12:03.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:87da0633:::157:head 2023-12-15T17:12:03.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d5ac3822:::158:head 2023-12-15T17:12:03.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3f20522d:::159:head 2023-12-15T17:12:03.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6ca26563:::160:head 2023-12-15T17:12:03.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:532ce135:::161:head 2023-12-15T17:12:03.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:c78863e6:::162:head 2023-12-15T17:12:04.854 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:14.799 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:04.857 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:14.803 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:04.879 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:14.824 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:04.910 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:14.855 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:04.911 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:14.857 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:04.917 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:14.856 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:04.918 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:14.858 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:04.933 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:14.879 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:04.939 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:14.879 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:15.419 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:25.365 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:15.421 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:25.366 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:15.437 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:25.383 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:15.481 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:25.420 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:15.482 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:25.422 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:15.485 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:25.431 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:15.486 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:25.432 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:15.499 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:25.438 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:15.561 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:25.507 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:35.599 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:45.544 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:35.607 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:45.552 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:35.645 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:45.591 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:35.661 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:45.601 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:35.663 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:45.603 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:35.669 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:45.614 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:35.675 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:45.620 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:35.691 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:45.630 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:35.714 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:45.660 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:55.609 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:05.554 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:55.666 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:05.611 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:55.700 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:05.646 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:55.721 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:05.660 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:55.721 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:05.667 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:55.723 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:05.669 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:55.725 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:05.664 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:55.757 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:05.697 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:12:55.762 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:05.707 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:00.543 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:10.488 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:00.545 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:10.491 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:00.563 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:10.508 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:00.588 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:10.528 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:00.591 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:10.531 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:00.593 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:10.539 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:00.595 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:10.541 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:00.606 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:10.545 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:00.612 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:10.558 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:15.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7462ddf6:::163:head 2023-12-15T17:13:15.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:5d165639:::164:head 2023-12-15T17:13:15.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f43765fc:::165:head 2023-12-15T17:13:15.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b4c720e9:::166:head 2023-12-15T17:13:15.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e694b040:::167:head 2023-12-15T17:13:15.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:afa38db2:::168:head 2023-12-15T17:13:15.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:77ba9f53:::169:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:87495034:::170:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7c96bf0e:::171:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:dbe346cc:::172:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:e943ec24:::173:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f97a9c0c:::174:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:6f26e74d:::175:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4f95e106:::176:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:0e6f2f8f:::177:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:05db05f1:::178:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:38a78d66:::179:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:d095610b:::180:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:a1a9d709:::181:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:1e5d39db:::182:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:f7df4fb9:::183:head 2023-12-15T17:13:15.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:03a7f161:::184:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ba70721e:::185:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:28e5662d:::186:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:973d52de:::187:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:4303eb1c:::188:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:b990b48e:::189:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:29b8165b:::190:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:3547f197:::191:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:7e260936:::192:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:1abec7b1:::193:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:10fdda93:::194:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:15817eea:::195:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:770bab57:::196:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:ed9e13e7:::197:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:71471a8f:::198:head 2023-12-15T17:13:15.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 201:10fb1d02:::199:head 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (22471 ms) 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,0 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first is 1702660342 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,0 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,0 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,1702660344,1702660345,0 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,1702660344,1702660345,0 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,1702660344,1702660345,0 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,1702660344,1702660345,1702660347,1702660348,0 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,1702660344,1702660345,1702660347,1702660348,0 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,1702660344,1702660345,1702660347,1702660348,0 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,1702660344,1702660345,1702660347,1702660348,1702660351,1702660352,0 2023-12-15T17:13:15.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,1702660344,1702660345,1702660347,1702660348,1702660351,1702660352,0 2023-12-15T17:13:15.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660342,1702660344,1702660345,1702660347,1702660348,1702660351,1702660352,0 2023-12-15T17:13:15.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660345,1702660347,1702660348,1702660351,1702660352,1702660354,1702660355,0 2023-12-15T17:13:15.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first now 1702660345, trimmed 2023-12-15T17:13:15.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22809 ms) 2023-12-15T17:13:15.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-12-15T17:13:15.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: foo0 2023-12-15T17:13:15.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-12-15T17:13:15.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (17621 ms) 2023-12-15T17:13:15.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-12-15T17:13:15.773 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:25.718 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:15.777 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:25.723 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:15.809 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:25.755 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:15.841 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:25.780 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:15.846 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:25.785 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:15.846 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:25.792 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:15.854 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:25.800 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:15.872 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:25.811 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:15.884 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:25.829 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:30.667 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:40.613 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:30.673 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:40.619 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:30.701 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:40.646 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:30.720 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:40.660 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:30.723 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:40.663 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:30.724 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:40.670 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:30.727 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:40.673 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:30.754 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:40.693 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:30.754 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:40.700 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:35.641 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:45.586 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:35.690 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:45.635 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:35.702 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:45.647 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:35.736 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:45.681 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:35.737 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:45.676 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:35.740 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:45.680 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:35.743 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:45.689 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:35.753 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:45.693 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:35.759 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:45.705 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:45.739 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:55.685 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:45.742 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:55.688 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:45.765 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:55.711 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:45.784 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:55.724 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:45.786 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:55.726 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:45.794 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:55.740 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:45.800 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:55.745 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:45.808 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:55.748 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:45.822 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:55.767 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:55.913 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:05.858 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:55.914 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:05.860 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:55.939 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:05.885 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:55.968 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:05.913 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:55.970 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:05.916 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:55.971 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:05.911 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:55.978 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:05.917 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:55.998 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:05.944 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:13:56.011 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:05.950 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:00.793 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:10.738 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:00.797 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:10.743 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:00.816 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:10.762 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:00.838 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:10.778 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:00.841 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:10.781 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:00.848 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:10.793 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:00.851 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:10.796 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:00.855 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:10.795 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:00.866 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:10.811 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:16.032 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:25.977 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:16.033 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:25.978 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:16.060 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:26.006 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:16.077 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:26.017 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:16.080 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:26.026 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:16.081 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:26.027 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:16.086 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:26.025 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:16.110 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:26.056 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:16.113 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:26.053 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:35.943 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:45.888 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:35.948 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:45.894 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:35.967 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:45.913 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:35.996 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:45.941 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:35.997 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:45.936 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:35.997 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:45.943 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:35.999 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:45.939 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:36.017 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:45.963 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:36.022 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:45.962 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:41.046 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:50.991 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:41.048 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:50.994 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:41.061 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:51.007 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:41.100 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:51.039 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:41.103 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:51.043 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:41.113 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:51.053 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:41.115 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:51.061 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:41.117 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:51.063 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:41.132 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:51.078 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:51.011 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:00.956 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:51.013 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:00.959 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:51.031 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:00.977 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:51.061 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:01.000 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:51.062 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:01.002 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:51.064 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:01.009 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:51.067 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:01.012 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:51.081 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:01.021 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:14:51.091 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:01.037 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:01.121 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:11.067 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:01.179 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:11.125 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:01.194 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:11.139 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:01.226 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:11.165 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:01.227 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:11.167 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:01.240 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:11.179 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:01.241 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:11.187 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:01.242 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:11.188 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:01.260 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:11.206 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:16.150 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:26.095 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:16.212 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:26.158 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:16.245 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:26.191 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:16.262 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:26.202 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:16.265 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:26.205 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:16.266 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:26.212 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:16.270 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:26.216 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:16.301 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:26.247 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:16.305 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:26.245 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:21.289 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:31.234 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:21.293 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:31.239 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:21.305 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:31.250 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:21.336 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:31.276 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:21.338 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:31.278 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:21.338 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:31.284 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:21.341 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:31.287 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:21.348 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:31.288 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:21.356 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:31.302 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:36.198 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:46.144 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:36.245 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:46.190 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:36.261 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:46.206 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:36.293 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:46.239 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:36.295 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:46.240 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:36.295 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:46.235 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:36.297 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:46.237 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:36.309 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:46.254 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:36.314 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:46.253 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:41.279 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:51.224 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:41.282 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:51.227 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:41.297 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:51.242 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:41.339 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:51.278 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:41.339 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:51.285 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:41.341 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:51.287 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:41.342 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:51.281 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:41.360 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:51.300 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:41.361 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:51.307 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:51.290 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:01.235 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:51.295 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:01.241 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:51.311 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:01.257 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:51.340 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:01.279 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:51.342 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:01.282 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:51.343 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:01.289 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:51.345 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:01.291 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:51.360 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:01.300 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:15:51.368 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:01.313 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:01.419 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:11.364 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:01.421 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:11.367 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:01.435 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:11.374 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:01.439 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:11.378 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:01.447 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:11.392 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:01.457 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:11.397 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:01.481 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:11.426 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:01.482 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:11.428 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:01.500 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:11.445 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:06.469 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:16.415 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:06.476 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:16.422 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:06.508 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:16.454 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:06.519 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:16.458 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:06.521 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:16.467 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:06.527 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:16.473 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:06.528 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:16.468 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:06.565 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:16.511 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:06.569 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:16.508 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:21.402 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:31.347 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:21.405 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:31.350 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:21.429 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:31.375 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:21.452 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:31.391 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:21.455 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:31.395 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:21.457 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:31.403 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:21.458 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:31.404 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:21.482 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:31.421 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:21.482 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:31.428 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:21.995 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:31.935 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:21.996 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:31.936 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:22.004 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:31.943 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:22.007 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:31.953 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:22.008 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:31.954 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:22.018 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:31.963 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:22.040 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:31.986 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:22.041 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:31.987 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:22.050 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:31.995 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:31.371 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-12-15T17:16:31.355+0000 24680640 -1 snap_mapper.add_oid found existing snaps mapped on 219:a18f4bb5:test-rados-api-smithi069-92319-80::foo:2, removing 2023-12-15T17:16:31.379 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-12-15T17:16:31.372+0000 2d692640 -1 snap_mapper.add_oid found existing snaps mapped on 219:a18f4bb5:test-rados-api-smithi069-92319-80::foo:2, removing 2023-12-15T17:16:38.909 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:48.854 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:38.963 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:48.908 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:38.967 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:48.907 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:38.969 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:48.909 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:38.995 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:48.934 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:39.012 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:48.955 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:39.033 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:48.978 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:39.038 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:48.983 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:39.069 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:49.014 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:41.569 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:51.514 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:41.616 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:51.556 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:41.617 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:51.557 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:41.634 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:51.574 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:41.639 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:51.585 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:41.662 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:51.608 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:41.700 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:51.646 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:41.704 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:51.649 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:41.717 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:51.662 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:56.569 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:06.514 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:56.573 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:06.519 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:56.602 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:06.548 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:56.619 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:06.558 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:56.621 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:06.561 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:56.623 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:06.569 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:56.625 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:06.571 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:56.649 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:06.588 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:16:56.654 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:06.599 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:01.578 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:11.523 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:01.580 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:11.526 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:01.595 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:11.541 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:01.624 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:11.564 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:01.626 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:11.572 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:01.627 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:11.567 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:01.632 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:11.578 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:01.640 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:11.580 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:01.646 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:11.592 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:06.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (20847 ms) 2023-12-15T17:17:06.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-12-15T17:17:06.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (24279 ms) 2023-12-15T17:17:06.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-12-15T17:17:06.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4464 ms) 2023-12-15T17:17:06.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-12-15T17:17:06.564 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (4588 ms) 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4513 ms) 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4944 ms) 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5531 ms) 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40729 ms) 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-12-15T17:17:06.565 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18858 ms) 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5357 ms) 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4294 ms) 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27257 ms) 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15957 ms) 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.566 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5372 ms) 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5028 ms) 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrubs... 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26202 ms) 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12046 ms) 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.567 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (10535 ms) 2023-12-15T17:17:06.568 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-12-15T17:17:06.568 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:17:06.568 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11391 ms) 2023-12-15T17:17:06.896 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:16.841 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:06.900 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:16.846 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:06.928 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:16.873 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:06.967 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:16.913 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:06.968 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:16.914 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:06.972 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:16.912 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:06.993 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:16.939 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:06.994 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:16.934 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:07.027 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:16.967 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:11.882 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:21.828 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:11.885 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:21.830 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:11.903 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:21.848 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:11.928 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:21.868 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:11.929 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:21.874 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:11.930 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:21.870 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:11.931 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:21.877 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:11.944 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:21.883 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:11.945 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:21.891 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:21.937 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:31.883 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:21.939 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:31.885 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:21.964 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:31.910 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:21.989 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:31.929 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:21.991 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:31.930 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:22.002 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:31.947 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:22.003 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:31.949 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:22.012 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:31.951 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:22.025 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:31.971 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:27.110 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:37.056 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:27.114 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:37.060 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:27.136 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:37.082 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:27.173 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:37.119 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:27.185 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:37.125 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:27.186 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:37.132 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:27.196 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:37.135 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:27.208 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:37.154 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:27.214 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:37.154 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.015 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:51.961 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.017 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:51.963 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.041 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:51.987 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.063 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:52.009 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.064 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:52.003 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.067 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:52.007 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.067 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:52.013 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.090 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:52.036 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.090 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:52.030 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.586 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:52.525 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.587 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:52.526 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.595 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:52.534 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.609 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:52.555 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.610 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:52.556 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.618 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:52.564 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.660 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:52.606 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.661 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:52.607 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:17:42.671 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:52.616 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:02.056 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:12.001 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:02.112 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:12.058 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:02.137 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:12.083 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:02.159 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:12.105 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:02.167 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:12.107 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:02.169 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:12.109 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:02.170 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:12.116 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:02.196 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:12.136 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:02.206 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:12.152 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:07.238 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:17.184 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:07.244 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:17.189 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:07.254 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:17.200 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:07.283 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:17.223 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:07.285 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:17.225 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:07.287 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:17.233 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:07.289 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:17.235 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:07.296 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:17.236 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:07.301 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:17.246 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:13.373 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-12-15T17:18:13.367+0000 2b68e640 -1 snap_mapper.add_oid found existing snaps mapped on 60:8851dedc:test-rados-api-smithi069-92319-97::foo:31, removing 2023-12-15T17:18:13.377 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-12-15T17:18:13.357+0000 24680640 -1 snap_mapper.add_oid found existing snaps mapped on 60:8851dedc:test-rados-api-smithi069-92319-97::foo:31, removing 2023-12-15T17:18:17.138 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:27.084 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:17.182 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:27.122 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:17.183 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:27.123 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:17.186 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:27.132 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:17.199 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:27.139 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:17.211 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:27.157 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:17.234 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:27.180 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:17.235 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:27.181 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:17.252 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:27.197 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:27.210 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:37.155 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:27.214 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:37.159 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:27.244 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:37.190 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:27.263 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:37.202 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:27.265 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:37.205 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:27.266 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:37.212 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:27.269 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:37.215 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:27.288 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:37.227 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:27.295 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:37.241 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:32.225 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:42.171 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:32.226 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:42.172 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:32.239 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:42.185 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:32.273 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:42.213 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:32.274 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:42.220 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:32.274 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:42.214 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:32.275 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:42.221 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:32.286 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:42.226 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:32.288 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:42.234 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:45.663 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:55.609 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:45.676 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:55.622 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:45.677 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:55.617 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:45.679 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:55.619 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:45.694 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:55.640 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:45.695 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:55.641 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:45.755 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:55.695 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:45.770 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:55.715 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:45.789 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:55.735 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.337 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:57.283 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.339 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:57.285 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.349 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:57.295 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.389 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:57.328 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.389 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:57.335 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.391 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:57.337 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.392 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:57.330 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.401 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:57.346 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.404 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:57.343 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.881 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:57.821 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.882 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:57.822 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.890 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:57.830 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.927 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:57.873 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.928 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:57.874 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.936 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:57.882 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.938 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:57.884 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.939 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:57.885 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:47.948 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:57.894 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:52.390 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:02.335 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:52.391 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:02.337 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:52.407 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:02.352 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:52.439 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:02.379 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:52.441 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:02.380 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:52.443 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:02.389 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:52.446 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:02.392 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:52.455 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:02.395 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:18:52.462 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:02.408 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:02.415 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:12.361 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:02.420 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:12.365 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:02.457 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:12.402 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:02.470 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:12.410 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:02.471 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:12.411 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:02.478 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:12.424 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:02.483 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:12.428 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:02.493 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:12.433 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:02.507 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:12.452 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:07.434 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:17.379 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:07.444 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:17.389 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:07.457 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:17.403 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:07.483 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:17.422 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:07.483 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:17.429 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:07.484 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:17.424 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:07.485 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:17.431 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:07.497 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:17.436 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:07.503 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:17.449 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:17.465 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:27.411 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:17.470 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:27.416 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:17.494 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:27.440 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:17.517 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:27.456 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:17.519 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:27.465 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:17.519 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:27.459 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:17.521 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:27.466 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:17.542 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:27.482 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:17.545 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:27.491 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:27.582 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:37.528 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:27.585 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:37.530 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:27.608 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:37.554 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:27.637 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:37.583 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:27.637 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:37.577 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:27.638 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:37.584 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:27.644 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:37.583 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:27.664 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:37.610 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:27.674 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:37.613 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:32.531 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:42.476 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:32.534 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:42.480 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:32.551 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:42.497 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:32.580 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:42.520 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:32.582 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:42.527 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:32.582 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:42.522 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:32.585 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:42.531 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:32.597 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:42.536 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:32.599 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:42.545 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:47.685 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:57.631 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:47.689 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:57.635 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:47.728 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:57.674 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:47.747 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:57.692 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:47.749 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:57.695 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:47.753 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:57.693 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:47.757 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:57.697 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:47.784 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:57.730 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:47.787 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:57.727 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:48.347 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:58.286 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:48.348 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:58.288 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:48.355 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:58.295 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:48.388 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:58.334 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:48.389 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:58.335 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:48.402 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:58.348 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:48.403 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:58.349 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:48.405 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:58.351 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:19:48.413 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:58.359 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:02.694 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:12.640 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:02.698 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:12.644 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:02.721 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:12.666 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:02.745 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:12.691 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:02.747 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:12.687 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:02.749 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:12.689 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:02.751 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:12.697 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:02.770 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:12.710 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:02.775 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:12.721 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:07.690 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:17.636 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:07.694 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:17.640 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:07.706 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:17.652 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:07.744 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:17.684 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:07.745 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:17.685 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:07.747 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:17.693 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:07.751 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:17.697 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:07.755 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:17.695 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:07.764 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:17.709 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:17.845 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:27.790 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:17.847 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:27.793 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:17.867 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:27.813 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:17.891 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:27.831 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:17.892 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:27.832 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:17.893 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:27.839 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:17.894 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:27.840 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:17.912 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:27.852 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:17.916 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:27.862 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:27.906 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:37.851 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:27.908 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:37.854 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:27.940 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:37.886 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:27.955 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:37.901 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:27.955 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:37.895 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:27.956 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:37.902 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:27.956 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:37.896 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:27.977 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:37.917 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:27.978 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:37.923 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:32.956 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:42.901 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:32.957 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:42.903 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:32.972 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:42.918 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:33.000 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:42.946 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:33.004 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:42.949 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:33.011 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:42.950 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:33.011 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:42.951 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:33.017 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:42.963 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:33.026 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:42.966 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:41.771 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:51.717 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:41.792 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:51.738 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:41.819 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:51.765 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:41.833 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:51.778 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:41.834 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:51.780 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:41.836 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:51.776 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:41.840 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:51.779 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:41.865 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:51.805 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:41.866 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:51.812 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:47.880 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:57.826 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:47.883 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:57.829 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:47.899 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:57.845 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:47.929 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:57.868 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:47.929 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:57.869 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:47.937 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:57.883 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:47.938 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:57.884 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:47.943 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:57.883 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:47.953 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:57.899 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:48.559 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:58.498 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:48.560 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:58.500 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:48.570 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:58.510 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:48.585 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:58.530 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:48.586 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:58.532 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:48.602 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:58.548 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:48.604 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:58.549 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:48.604 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:58.550 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:20:48.614 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:58.560 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:01.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-12-15T17:21:01.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:21:01.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (11519 ms) 2023-12-15T17:21:01.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-12-15T17:21:01.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:21:01.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8166 ms) 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5764 ms) 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (27536 ms) 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15556 ms) 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (13291 ms) 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-12-15T17:21:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000010 69 73 |is| 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000012 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (13227 ms) 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7040 ms) 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (857737 ms total) 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1496 ms) 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10592 ms) 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-12-15T17:21:01.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19715 ms) 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (46 ms) 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (31849 ms total) 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7921 ms) 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8600 ms) 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrub... 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (26438 ms) 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10415 ms) 2023-12-15T17:21:01.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-12-15T17:21:01.495 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8593 ms) 2023-12-15T17:21:01.495 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-12-15T17:21:01.495 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8699 ms) 2023-12-15T17:21:01.495 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-12-15T17:21:01.495 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11122 ms) 2023-12-15T17:21:01.495 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-12-15T17:21:03.149 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:13.094 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:03.152 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:13.097 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:03.190 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:13.136 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:03.203 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:13.142 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:03.206 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:13.146 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:03.218 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:13.164 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:03.221 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:13.167 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:03.247 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:13.186 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:03.266 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:13.212 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:08.103 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:18.049 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:08.107 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:18.053 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:08.125 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:18.071 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:08.157 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:18.097 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:08.158 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:18.098 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:08.165 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:18.111 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:08.170 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:18.116 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:08.174 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:18.114 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:08.185 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:18.130 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:18.133 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:28.078 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:18.190 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:28.136 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:18.198 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:28.138 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:18.214 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:28.154 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:18.220 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:28.166 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:18.242 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:28.182 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:18.254 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:28.200 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:18.287 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:28.233 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:18.325 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:28.271 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:28.077 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:38.023 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:28.082 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:38.028 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:28.106 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:38.051 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:28.127 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:38.067 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:28.128 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:38.074 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:28.129 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:38.069 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:28.131 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:38.077 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:28.151 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:38.090 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:28.155 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:38.101 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:38.162 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:48.107 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:38.209 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:48.148 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:38.210 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:48.149 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:38.214 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:48.160 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:38.233 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:48.172 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:38.238 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:48.184 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:38.268 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:48.214 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:38.271 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:48.217 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:38.294 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:48.240 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:48.160 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:58.105 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:48.164 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:58.109 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:48.188 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:58.133 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:48.209 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:58.148 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:48.211 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:58.151 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:48.211 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:58.157 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:48.214 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:58.160 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:48.233 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:58.173 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:48.236 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:58.182 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:53.282 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:03.228 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:53.286 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:03.232 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:53.304 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:03.250 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:53.332 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:03.272 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:53.333 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:03.273 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:53.341 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:03.287 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:53.348 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:03.288 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:53.353 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:03.299 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:21:53.369 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:03.315 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:08.353 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:18.299 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:08.356 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:18.302 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:08.383 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:18.329 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:08.406 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:18.352 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:08.407 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:18.353 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:08.417 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:18.356 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:08.418 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:18.358 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:08.448 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:18.394 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:08.449 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:18.388 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:13.243 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:23.188 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:13.247 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:23.193 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:13.262 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:23.207 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:13.293 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:23.233 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:13.294 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:23.240 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:13.295 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:23.235 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:13.296 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:23.242 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:13.308 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:23.248 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:13.310 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:23.256 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:28.287 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:38.232 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:28.290 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:38.236 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:28.310 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:38.256 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:28.334 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:38.280 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:28.337 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:38.277 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:28.339 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:38.285 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:28.339 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:38.279 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:28.357 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:38.297 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:28.364 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:38.310 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:33.433 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:43.379 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:33.437 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:43.383 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:33.440 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:43.380 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:33.442 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:43.382 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:33.443 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:43.389 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:33.445 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:43.391 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:33.512 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:43.452 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:33.539 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:43.485 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:33.568 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:43.513 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:45.842 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:55.787 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:45.847 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:55.793 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:45.878 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:55.823 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:45.897 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:55.837 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:45.898 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:55.838 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:45.914 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:55.856 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:45.914 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:55.858 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:45.931 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:55.870 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:45.943 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:55.889 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:48.346 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:58.291 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:48.348 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:58.293 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:48.364 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:58.309 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:48.390 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:58.335 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:48.395 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:58.335 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:48.395 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:58.341 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:48.397 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:58.337 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:48.407 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:58.347 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:48.408 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:58.354 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:48.999 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:58.939 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:49.000 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:58.940 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:49.008 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:58.948 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:49.125 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:59.071 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:49.126 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:59.072 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:49.138 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:59.084 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:49.172 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:59.117 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:49.173 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:59.119 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:22:49.185 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:59.131 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:03.400 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:13.346 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:03.403 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:13.349 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:03.425 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:13.370 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:03.453 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:13.393 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:03.454 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:13.394 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:03.456 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:13.402 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:03.460 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:13.406 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:03.475 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:13.415 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:03.481 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:13.427 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:08.371 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:18.317 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:08.416 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:18.355 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:08.417 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:18.357 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:08.420 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:18.366 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:08.432 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:18.372 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:08.438 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:18.384 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:08.467 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:18.413 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:08.473 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:18.419 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:08.487 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:18.432 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:18.453 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:28.399 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:18.455 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:28.401 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:18.475 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:28.420 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:18.500 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:28.440 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:18.502 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:28.442 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:18.502 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:28.448 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:18.507 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:28.453 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:18.522 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:28.461 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:18.526 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:28.472 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:33.496 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:43.441 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:33.499 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:43.445 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:33.521 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:43.467 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:33.548 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:43.493 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:33.551 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:43.497 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:33.552 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:43.492 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:33.554 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:43.493 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:33.574 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:43.519 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:33.575 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:43.515 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:34.249 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:44.188 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:34.252 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:44.192 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:34.263 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:44.203 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:34.360 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:44.306 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:34.361 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:44.306 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:34.362 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:44.307 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:34.362 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:44.308 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:34.370 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:44.315 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:34.371 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:44.317 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:48.518 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:58.464 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:48.584 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:58.530 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:48.586 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:58.532 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:48.587 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:58.526 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:48.589 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:58.535 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:48.591 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:58.531 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:48.633 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:58.579 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:48.634 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:58.574 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:48.657 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:58.603 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:53.573 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:03.518 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:53.577 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:03.523 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:53.593 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:03.539 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:53.621 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:03.561 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:53.624 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:03.564 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:53.626 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:03.572 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:53.629 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:03.575 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:53.639 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:03.579 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:53.644 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:03.589 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:23:58.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8812 ms) 2023-12-15T17:23:58.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-12-15T17:23:58.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14144 ms) 2023-12-15T17:23:58.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-12-15T17:23:58.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8706 ms) 2023-12-15T17:23:58.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-12-15T17:23:58.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15483 ms) 2023-12-15T17:23:58.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-12-15T17:23:58.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (9835 ms) 2023-12-15T17:23:58.320 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8569 ms) 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: hmm, no HitSet yet 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (10065 ms) 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,0 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first is 1702660937 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,0 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,0 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,1702660939,1702660940,0 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,1702660939,1702660940,0 2023-12-15T17:23:58.321 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,1702660939,1702660940,0 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,1702660939,1702660940,1702660942,1702660943,0 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,1702660939,1702660940,1702660942,1702660943,0 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,1702660939,1702660940,1702660942,1702660943,0 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,1702660939,1702660940,1702660942,1702660943,1702660945,1702660946,0 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,1702660939,1702660940,1702660942,1702660943,1702660945,1702660946,0 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660937,1702660939,1702660940,1702660942,1702660943,1702660945,1702660946,0 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1702660940,1702660942,1702660943,1702660945,1702660946,1702660948,1702660949,0 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first now 1702660940, trimmed 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22855 ms) 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: foo0 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (17833 ms) 2023-12-15T17:23:58.322 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (21197 ms) 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (25108 ms) 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (6591 ms) 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4258 ms) 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4506 ms) 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4433 ms) 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (264184 ms total) 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1237020 ms total) 2023-12-15T17:23:58.323 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91986 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91986 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92196 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92196 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92706 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92706 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92398 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92398 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92252 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92252 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92067 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92067 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92339 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92339 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92092 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92092 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92579 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92579 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91958 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91958 2023-12-15T17:23:58.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92023 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92023 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92632 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92632 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91949 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91949 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91971 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91971 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92207 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92207 2023-12-15T17:23:58.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92309 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92309 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92224 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92224 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92688 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92688 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92556 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92556 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92154 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92154 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92438 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92438 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=91942 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 91942 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92001 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92001 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92374 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92374 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92043 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92043 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92465 2023-12-15T17:23:58.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92465 2023-12-15T17:23:58.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-12-15T17:23:58.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=92499 2023-12-15T17:23:58.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 92499 2023-12-15T17:23:58.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ exit 0 2023-12-15T17:23:58.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ cleanup 2023-12-15T17:23:58.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ pkill -P 91937 2023-12-15T17:23:58.341 INFO:tasks.workunit.client.0.smithi069.stderr:+ true 2023-12-15T17:23:58.342 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-15T17:23:58.342 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-15T17:23:58.371 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-12-15T17:23:58.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-15T17:23:58.683 DEBUG:teuthology.parallel:result is None 2023-12-15T17:23:58.684 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-15T17:23:58.711 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-15T17:23:58.712 DEBUG:teuthology.orchestra.run.smithi069:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-15T17:23:58.773 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-15T17:23:58.773 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-15T17:23:58.782 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-15T17:23:58.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T17:23:58.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.910+0000 7f2ad11d7640 10 monclient: get_monmap_and_config 2023-12-15T17:23:58.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.910+0000 7f2ad11d7640 10 monclient: build_initial_monmap 2023-12-15T17:23:58.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.910+0000 7f2ad11d7640 10 monclient: monmap: 2023-12-15T17:23:58.911 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:23:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:23:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:23:58.911618+0000 2023-12-15T17:23:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:23:58.911618+0000 2023-12-15T17:23:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:23:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:23:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:23:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:23:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:23:58.912 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:23:58.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:58.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.910+0000 7f2ad11d7640 1 Processor -- start 2023-12-15T17:23:58.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.910+0000 7f2ad11d7640 1 -- start start 2023-12-15T17:23:58.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.910+0000 7f2ad11d7640 10 monclient: init 2023-12-15T17:23:58.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 10 monclient: _reopen_session rank -1 2023-12-15T17:23:58.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-12-15T17:23:58.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2acc0a8a30 0x7f2acc1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 10 monclient(hunting): picked mon.noname-a con 0x7f2acc0a8a30 addr v2:172.21.15.69:3300/0 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2acc0a8e10 0x7f2acc142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 10 monclient(hunting): picked mon.noname-c con 0x7f2acc0a8e10 addr v2:172.21.15.192:3300/0 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2acc1454a0 addr v1:172.21.15.192:6789/0 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2acc0aac10 con 0x7f2acc1454a0 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2acc0aaa10 con 0x7f2acc0a8a30 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2acc0aaef0 con 0x7f2acc0a8e10 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 10 monclient(hunting): _renew_subs 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2ad11d7640 10 monclient(hunting): authenticate will time out at 2743.227786s 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.911+0000 7f2acad9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2acc0a8a30 0x7f2acc1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2aca59c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2acc0a8e10 0x7f2acc142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:58.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2acad9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2acc0a8a30 0x7f2acc1424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45260/0 (socket says 172.21.15.69:45260) 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2acad9d640 1 -- 172.21.15.69:0/840408328 learned_addr learned my addr 172.21.15.69:0/840408328 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2acad9d640 10 monclient(hunting): get_auth_request con 0x7f2acc0a8a30 auth_method 0 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2acad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2acad9d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2acad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2aca59c640 10 monclient(hunting): get_auth_request con 0x7f2acc0a8e10 auth_method 0 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2aca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2aca59c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2aca59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2acad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2acad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.912+0000 7f2acad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.913+0000 7f2aca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.913+0000 7f2aca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:23:58.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.913+0000 7f2aca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:23:58.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.914+0000 7f2acad9d640 10 monclient(hunting): handle_auth_done global_id 16047 payload 274 2023-12-15T17:23:58.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.914+0000 7f2acad9d640 1 -- 172.21.15.69:0/840408328 >> v2:172.21.15.192:3300/0 conn(0x7f2acc0a8e10 msgr2=0x7f2acc142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:23:58.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.914+0000 7f2acad9d640 1 --2- 172.21.15.69:0/840408328 >> v2:172.21.15.192:3300/0 conn(0x7f2acc0a8e10 0x7f2acc142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:58.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.914+0000 7f2acad9d640 1 -- 172.21.15.69:0/840408328 >> v1:172.21.15.192:6789/0 conn(0x7f2acc1454a0 legacy=0x7f2acc1458f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:23:58.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.914+0000 7f2acad9d640 10 monclient: _finish_hunting 0 2023-12-15T17:23:58.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.914+0000 7f2acad9d640 1 monclient: found mon.noname-a 2023-12-15T17:23:58.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.914+0000 7f2acad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:23:58.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.914+0000 7f2acad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:23:58.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.914+0000 7f2acad9d640 1 -- 172.21.15.69:0/840408328 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc0a97f0 con 0x7f2acc0a8a30 2023-12-15T17:23:58.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.916+0000 7f2acad9d640 1 --2- 172.21.15.69:0/840408328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2acc0a8a30 0x7f2acc1424e0 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0b87d0 tx=0x7f2abc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4260fea8d2965db server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:23:58.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 1 -- 172.21.15.69:0/840408328 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc02cb20 con 0x7f2acc0a8a30 2023-12-15T17:23:58.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:23:58.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:23:58.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 10 monclient: dump: 2023-12-15T17:23:58.925 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 10 monclient: _finish_auth 0 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 10 monclient: _check_auth_tickets 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:23:58.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 1 -- 172.21.15.69:0/840408328 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc02cc80 con 0x7f2acc0a8a30 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 1 -- 172.21.15.69:0/840408328 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc0343f0 con 0x7f2acc0a8a30 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ad11d7640 5 monclient: authenticate success, global_id 16047 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ad11d7640 10 monclient: get_monmap_and_config success 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ad11d7640 4 set_mon_vals no callback set 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.919+0000 7f2ac9d9b640 10 monclient: dump: 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:23:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.920+0000 7f2ad11d7640 10 monclient: shutdown 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.920+0000 7f2ad11d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.920+0000 7f2ad11d7640 1 -- 172.21.15.69:0/840408328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2acc0a8a30 msgr2=0x7f2acc1424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.920+0000 7f2ad11d7640 1 --2- 172.21.15.69:0/840408328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2acc0a8a30 0x7f2acc1424e0 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0b87d0 tx=0x7f2abc02bf80 comp rx=0 tx=0).stop 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.920+0000 7f2ad11d7640 1 -- 172.21.15.69:0/840408328 shutdown_connections 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.920+0000 7f2ad11d7640 1 --2- 172.21.15.69:0/840408328 >> v2:172.21.15.192:3300/0 conn(0x7f2acc0a8e10 0x7f2acc142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.920+0000 7f2ad11d7640 1 --2- 172.21.15.69:0/840408328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2acc0a8a30 0x7f2acc1424e0 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.920+0000 7f2ad11d7640 1 -- 172.21.15.69:0/840408328 >> 172.21.15.69:0/840408328 conn(0x7f2acc061a30 msgr2=0x7f2acc0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.920+0000 7f2ad11d7640 1 -- 172.21.15.69:0/840408328 shutdown_connections 2023-12-15T17:23:58.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.920+0000 7f2ad11d7640 1 -- 172.21.15.69:0/840408328 wait complete. 2023-12-15T17:23:58.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.921+0000 7f2ad11d7640 10 monclient: build_initial_monmap 2023-12-15T17:23:58.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.921+0000 7f2ad11d7640 10 monclient: monmap: 2023-12-15T17:23:58.932 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:23:58.932 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:23:58.932 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:23:58.932 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:23:58.932 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:23:58.932 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:58.932 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:23:58.932 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:23:58.932 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.921+0000 7f2ad11d7640 1 librados: starting msgr at 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.921+0000 7f2ad11d7640 1 librados: starting objecter 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.921+0000 7f2ad11d7640 1 Processor -- start 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 1 -- start start 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 1 librados: setting wanted keys 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 1 librados: calling monclient init 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 10 monclient: init 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 10 monclient: _reopen_session rank -1 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2acc0a8a30 0x7f2acc1d75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 10 monclient(hunting): picked mon.noname-a con 0x7f2acc0a8a30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2acc0a8e10 0x7f2acc0aec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 10 monclient(hunting): picked mon.noname-b con 0x7f2acc0a8e10 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2acc1454a0 0x7f2acc0af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 10 monclient(hunting): picked mon.noname-c con 0x7f2acc1454a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0b10e0 con 0x7f2acc0a8a30 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc1d7fa0 con 0x7f2acc0a8e10 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 10 monclient(hunting): start opening mon connection 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0ae0b0 con 0x7f2acc1454a0 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 10 monclient(hunting): _renew_subs 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.922+0000 7f2ad11d7640 10 monclient(hunting): authenticate will time out at 2743.238963s 2023-12-15T17:23:58.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2acc0a8a30 0x7f2acc1d75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:58.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2aca59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2acc0a8e10 0x7f2acc0aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:58.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acb59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2acc1454a0 0x7f2acc0af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:58.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2acc0a8a30 0x7f2acc1d75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45270/0 (socket says 172.21.15.69:45270) 2023-12-15T17:23:58.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2aca59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2acc0a8e10 0x7f2acc0aec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:43822/0 (socket says 172.21.15.69:43822) 2023-12-15T17:23:58.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acad9d640 1 -- 172.21.15.69:0/1606315317 learned_addr learned my addr 172.21.15.69:0/1606315317 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acad9d640 10 monclient(hunting): get_auth_request con 0x7f2acc0a8a30 auth_method 0 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acad9d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2aca59c640 10 monclient(hunting): get_auth_request con 0x7f2acc0a8e10 auth_method 0 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2aca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2aca59c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2aca59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acb59e640 10 monclient(hunting): get_auth_request con 0x7f2acc1454a0 auth_method 0 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acb59e640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:23:58.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acb59e640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2aca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2aca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2aca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.923+0000 7f2acad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.924+0000 7f2acb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.924+0000 7f2acb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.924+0000 7f2acb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.926+0000 7f2aca59c640 10 monclient(hunting): handle_auth_done global_id 21409 payload 824 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.926+0000 7f2aca59c640 1 -- 172.21.15.69:0/1606315317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2acc1454a0 msgr2=0x7f2acc0af1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.926+0000 7f2aca59c640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2acc1454a0 0x7f2acc0af1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.926+0000 7f2aca59c640 1 -- 172.21.15.69:0/1606315317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2acc0a8a30 msgr2=0x7f2acc1d75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:23:58.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.926+0000 7f2aca59c640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2acc0a8a30 0x7f2acc1d75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:58.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.926+0000 7f2aca59c640 10 monclient: _finish_hunting 0 2023-12-15T17:23:58.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.926+0000 7f2aca59c640 1 monclient: found mon.noname-b 2023-12-15T17:23:58.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.926+0000 7f2aca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:23:58.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.926+0000 7f2aca59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:23:58.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.926+0000 7f2aca59c640 1 -- 172.21.15.69:0/1606315317 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc009b30 con 0x7f2acc0a8e10 2023-12-15T17:23:58.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.928+0000 7f2aca59c640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2acc0a8e10 0x7f2acc0aec80 secure :-1 s=READY pgs=5098 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000e880 tx=0x7f2ac000ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:23:58.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.928+0000 7f2ab37fe640 10 client.?.objecter ms_handle_connect 0x7f2acc0a8e10 2023-12-15T17:23:58.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.928+0000 7f2ab37fe640 10 client.?.objecter resend_mon_ops 2023-12-15T17:23:58.939 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 1 -- 172.21.15.69:0/1606315317 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000c940 con 0x7f2acc0a8e10 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 10 monclient: dump: 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:23:58.940 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 10 monclient: _finish_auth 0 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 10 monclient: _check_auth_tickets 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 1 -- 172.21.15.69:0/1606315317 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac000caa0 con 0x7f2acc0a8e10 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ad11d7640 5 monclient: authenticate success, global_id 21409 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ad11d7640 10 monclient: _renew_subs 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ad11d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ad11d7640 1 -- 172.21.15.69:0/1606315317 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc1d7b10 con 0x7f2acc0a8e10 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 1 -- 172.21.15.69:0/1606315317 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000ce50 con 0x7f2acc0a8e10 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:23:58.941 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ab37fe640 10 monclient: dump: 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ac959a640 4 set_mon_vals no callback set 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ad11d7640 10 client.21409.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ad11d7640 10 monclient: _renew_subs 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ad11d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:23:58.942 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ad11d7640 1 -- 172.21.15.69:0/1606315317 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc0af8e0 con 0x7f2acc0a8e10 2023-12-15T17:23:58.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ad11d7640 1 librados: init done 2023-12-15T17:23:58.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.932+0000 7f2ad11d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:23:58.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.933+0000 7f2ad11d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:23:58.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.933+0000 7f2ad11d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:23:58.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:58.933+0000 7f2ad11d7640 1 -- 172.21.15.69:0/1606315317 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc067ac0 con 0x7f2acc0a8e10 2023-12-15T17:23:59.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.187+0000 7f2ab37fe640 1 -- 172.21.15.69:0/1606315317 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2ac0013470 con 0x7f2acc0a8e10 2023-12-15T17:23:59.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.187+0000 7f2ab37fe640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2ab806cba0 0x7f2ab806f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:59.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.187+0000 7f2acad9d640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2ab806cba0 0x7f2ab806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:59.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.188+0000 7f2acad9d640 10 monclient: get_auth_request con 0x7f2ab806cba0 auth_method 0 2023-12-15T17:23:59.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.188+0000 7f2ab37fe640 10 client.21409.objecter ms_handle_connect 0x7f2ab806cba0 2023-12-15T17:23:59.189 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.188+0000 7f2acad9d640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2ab806cba0 0x7f2ab806f060 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2abc02bfc0 tx=0x7f2abc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:23:59.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.192+0000 7f2ab37fe640 1 -- 172.21.15.69:0/1606315317 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 534..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7f2ac0058e50 con 0x7f2acc0a8e10 2023-12-15T17:23:59.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.192+0000 7f2ab37fe640 10 client.21409.objecter ms_dispatch 0x7f2acc000b90 osd_map(908..908 src has 534..908) v4 2023-12-15T17:23:59.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.192+0000 7f2ab37fe640 3 client.21409.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:23:59.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.192+0000 7f2ab37fe640 3 client.21409.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:23:59.193 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.192+0000 7f2ab37fe640 20 client.21409.objecter dump_active .. 0 homeless 2023-12-15T17:23:59.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.201+0000 7f2ab37fe640 1 -- 172.21.15.69:0/1606315317 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2ac0055190 con 0x7f2acc0a8e10 2023-12-15T17:23:59.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.201+0000 7f2ab37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:23:59.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.201+0000 7f2ab37fe640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:23:59.370 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.369+0000 7f2ad11d7640 1 -- 172.21.15.69:0/1606315317 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2acc146000 con 0x7f2ab806cba0 2023-12-15T17:23:59.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.374+0000 7f2ab37fe640 1 -- 172.21.15.69:0/1606315317 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151911 (secure 0 0 0) 0x7f2abc045090 con 0x7f2ab806cba0 2023-12-15T17:23:59.377 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:23:59.378 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-12-15T17:23:59.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.376+0000 7f2ad11d7640 10 librados: watch_flush enter 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.376+0000 7f2ad11d7640 10 librados: watch_flush exit 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.376+0000 7f2ad11d7640 20 client.21409.objecter shutdown clearing up homeless session... 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.376+0000 7f2ad11d7640 10 client.21409.objecter successfully canceled tick 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.376+0000 7f2ad11d7640 1 -- 172.21.15.69:0/1606315317 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2ab806cba0 msgr2=0x7f2ab806f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.376+0000 7f2ad11d7640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2ab806cba0 0x7f2ab806f060 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2abc02bfc0 tx=0x7f2abc030000 comp rx=0 tx=0).stop 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.376+0000 7f2ad11d7640 10 monclient: shutdown 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.376+0000 7f2ad11d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.376+0000 7f2ad11d7640 1 -- 172.21.15.69:0/1606315317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2acc0a8e10 msgr2=0x7f2acc0aec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.376+0000 7f2ad11d7640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2acc0a8e10 0x7f2acc0aec80 secure :-1 s=READY pgs=5098 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000e880 tx=0x7f2ac000ed40 comp rx=0 tx=0).stop 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.377+0000 7f2ad11d7640 1 -- 172.21.15.69:0/1606315317 shutdown_connections 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.377+0000 7f2ad11d7640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f2ab806cba0 0x7f2ab806f060 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.377+0000 7f2ad11d7640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2acc1454a0 0x7f2acc0af1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.377+0000 7f2ad11d7640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2acc0a8e10 0x7f2acc0aec80 unknown :-1 s=CLOSED pgs=5098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.377+0000 7f2ad11d7640 1 --2- 172.21.15.69:0/1606315317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2acc0a8a30 0x7f2acc1d75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.377+0000 7f2ad11d7640 1 -- 172.21.15.69:0/1606315317 >> 172.21.15.69:0/1606315317 conn(0x7f2acc061a30 msgr2=0x7f2acc140100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:23:59.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.377+0000 7f2ad11d7640 1 -- 172.21.15.69:0/1606315317 shutdown_connections 2023-12-15T17:23:59.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.377+0000 7f2ad11d7640 1 -- 172.21.15.69:0/1606315317 wait complete. 2023-12-15T17:23:59.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.377+0000 7f2ad11d7640 1 librados: shutdown 2023-12-15T17:23:59.390 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1578,"stamp":"2023-12-15T17:23:58.565235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":570548,"kb_used_data":44848,"kb_used_omap":12,"kb_used_meta":525555,"kb_avail":754404172,"statfs":{"total":773094113280,"available":772509872128,"internally_reserved":0,"allocated":45924352,"data_stored":24850649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":538168880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":418,"apply_latency_ms":418,"commit_latency_ns":418000000,"apply_latency_ns":418000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.547303"},"pg_stats":[{"pgid":"7.1b","version":"0'0","reported_seq":864,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.792231+0000","last_change":"2023-12-15T17:03:26.413418+0000","last_active":"2023-12-15T17:23:51.792231+0000","last_peered":"2023-12-15T17:23:51.792231+0000","last_clean":"2023-12-15T17:23:51.792231+0000","last_became_active":"2023-12-15T17:03:26.401879+0000","last_became_peered":"2023-12-15T17:03:26.401879+0000","last_unstale":"2023-12-15T17:23:51.792231+0000","last_undegraded":"2023-12-15T17:23:51.792231+0000","last_fullsized":"2023-12-15T17:23:51.792231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:25:02.124791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.128322+0000","last_change":"2023-12-15T17:03:43.168165+0000","last_active":"2023-12-15T17:23:56.128322+0000","last_peered":"2023-12-15T17:23:56.128322+0000","last_clean":"2023-12-15T17:23:56.128322+0000","last_became_active":"2023-12-15T17:03:43.159317+0000","last_became_peered":"2023-12-15T17:03:43.159317+0000","last_unstale":"2023-12-15T17:23:56.128322+0000","last_undegraded":"2023-12-15T17:23:56.128322+0000","last_fullsized":"2023-12-15T17:23:56.128322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:28:01.151029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.123155+0000","last_change":"2023-12-15T17:03:27.146666+0000","last_active":"2023-12-15T17:23:56.123155+0000","last_peered":"2023-12-15T17:23:56.123155+0000","last_clean":"2023-12-15T17:23:56.123155+0000","last_became_active":"2023-12-15T17:03:27.137993+0000","last_became_peered":"2023-12-15T17:03:27.137993+0000","last_unstale":"2023-12-15T17:23:56.123155+0000","last_undegraded":"2023-12-15T17:23:56.123155+0000","last_fullsized":"2023-12-15T17:23:56.123155+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:13:48.832295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":864,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.774897+0000","last_change":"2023-12-15T17:03:32.164840+0000","last_active":"2023-12-15T17:23:51.774897+0000","last_peered":"2023-12-15T17:23:51.774897+0000","last_clean":"2023-12-15T17:23:51.774897+0000","last_became_active":"2023-12-15T17:03:32.155432+0000","last_became_peered":"2023-12-15T17:03:32.155432+0000","last_unstale":"2023-12-15T17:23:51.774897+0000","last_undegraded":"2023-12-15T17:23:51.774897+0000","last_fullsized":"2023-12-15T17:23:51.774897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:29:00.934002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":845,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-12-15T17:23:55.320927+0000","last_change":"2023-12-15T17:03:32.049516+0000","last_active":"2023-12-15T17:23:55.320927+0000","last_peered":"2023-12-15T17:23:55.320927+0000","last_clean":"2023-12-15T17:23:55.320927+0000","last_became_active":"2023-12-15T17:03:32.040162+0000","last_became_peered":"2023-12-15T17:03:32.040162+0000","last_unstale":"2023-12-15T17:23:55.320927+0000","last_undegraded":"2023-12-15T17:23:55.320927+0000","last_fullsized":"2023-12-15T17:23:55.320927+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:24:16.619046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":854,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.908823+0000","last_change":"2023-12-15T17:03:29.590131+0000","last_active":"2023-12-15T17:23:56.908823+0000","last_peered":"2023-12-15T17:23:56.908823+0000","last_clean":"2023-12-15T17:23:56.908823+0000","last_became_active":"2023-12-15T17:03:29.584729+0000","last_became_peered":"2023-12-15T17:03:29.584729+0000","last_unstale":"2023-12-15T17:23:56.908823+0000","last_undegraded":"2023-12-15T17:23:56.908823+0000","last_fullsized":"2023-12-15T17:23:56.908823+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:25:55.347921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":867,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.651419+0000","last_change":"2023-12-15T17:03:27.355790+0000","last_active":"2023-12-15T17:23:51.651419+0000","last_peered":"2023-12-15T17:23:51.651419+0000","last_clean":"2023-12-15T17:23:51.651419+0000","last_became_active":"2023-12-15T17:03:27.349390+0000","last_became_peered":"2023-12-15T17:03:27.349390+0000","last_unstale":"2023-12-15T17:23:51.651419+0000","last_undegraded":"2023-12-15T17:23:51.651419+0000","last_fullsized":"2023-12-15T17:23:51.651419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:50:15.500629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":849,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.754893+0000","last_change":"2023-12-15T17:03:41.190972+0000","last_active":"2023-12-15T17:23:56.754893+0000","last_peered":"2023-12-15T17:23:56.754893+0000","last_clean":"2023-12-15T17:23:56.754893+0000","last_became_active":"2023-12-15T17:03:41.182883+0000","last_became_peered":"2023-12-15T17:03:41.182883+0000","last_unstale":"2023-12-15T17:23:56.754893+0000","last_undegraded":"2023-12-15T17:23:56.754893+0000","last_fullsized":"2023-12-15T17:23:56.754893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T00:04:20.611993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":852,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.809588+0000","last_change":"2023-12-15T17:03:27.568351+0000","last_active":"2023-12-15T17:23:56.809588+0000","last_peered":"2023-12-15T17:23:56.809588+0000","last_clean":"2023-12-15T17:23:56.809588+0000","last_became_active":"2023-12-15T17:03:27.563090+0000","last_became_peered":"2023-12-15T17:03:27.563090+0000","last_unstale":"2023-12-15T17:23:56.809588+0000","last_undegraded":"2023-12-15T17:23:56.809588+0000","last_fullsized":"2023-12-15T17:23:56.809588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:48:11.443821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":843,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.866025+0000","last_change":"2023-12-15T17:03:26.203421+0000","last_active":"2023-12-15T17:23:51.866025+0000","last_peered":"2023-12-15T17:23:51.866025+0000","last_clean":"2023-12-15T17:23:51.866025+0000","last_became_active":"2023-12-15T17:03:26.168812+0000","last_became_peered":"2023-12-15T17:03:26.168812+0000","last_unstale":"2023-12-15T17:23:51.866025+0000","last_undegraded":"2023-12-15T17:23:51.866025+0000","last_fullsized":"2023-12-15T17:23:51.866025+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:30:33.793828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":849,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.744754+0000","last_change":"2023-12-15T17:03:33.666043+0000","last_active":"2023-12-15T17:23:56.744754+0000","last_peered":"2023-12-15T17:23:56.744754+0000","last_clean":"2023-12-15T17:23:56.744754+0000","last_became_active":"2023-12-15T17:03:33.660278+0000","last_became_peered":"2023-12-15T17:03:33.660278+0000","last_unstale":"2023-12-15T17:23:56.744754+0000","last_undegraded":"2023-12-15T17:23:56.744754+0000","last_fullsized":"2023-12-15T17:23:56.744754+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:21:15.535555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":852,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.757186+0000","last_change":"2023-12-15T17:03:29.499597+0000","last_active":"2023-12-15T17:23:56.757186+0000","last_peered":"2023-12-15T17:23:56.757186+0000","last_clean":"2023-12-15T17:23:56.757186+0000","last_became_active":"2023-12-15T17:03:29.479817+0000","last_became_peered":"2023-12-15T17:03:29.479817+0000","last_unstale":"2023-12-15T17:23:56.757186+0000","last_undegraded":"2023-12-15T17:23:56.757186+0000","last_fullsized":"2023-12-15T17:23:56.757186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:29:30.677748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":865,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.794565+0000","last_change":"2023-12-15T17:03:27.798402+0000","last_active":"2023-12-15T17:23:51.794565+0000","last_peered":"2023-12-15T17:23:51.794565+0000","last_clean":"2023-12-15T17:23:51.794565+0000","last_became_active":"2023-12-15T17:03:27.764430+0000","last_became_peered":"2023-12-15T17:03:27.764430+0000","last_unstale":"2023-12-15T17:23:51.794565+0000","last_undegraded":"2023-12-15T17:23:51.794565+0000","last_fullsized":"2023-12-15T17:23:51.794565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:50:51.103963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":882,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.172032+0000","last_change":"2023-12-15T17:02:01.284924+0000","last_active":"2023-12-15T17:23:56.172032+0000","last_peered":"2023-12-15T17:23:56.172032+0000","last_clean":"2023-12-15T17:23:56.172032+0000","last_became_active":"2023-12-15T17:02:01.275938+0000","last_became_peered":"2023-12-15T17:02:01.275938+0000","last_unstale":"2023-12-15T17:23:56.172032+0000","last_undegraded":"2023-12-15T17:23:56.172032+0000","last_fullsized":"2023-12-15T17:23:56.172032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:11:11.381089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":868,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.648118+0000","last_change":"2023-12-15T17:03:29.741478+0000","last_active":"2023-12-15T17:23:51.648118+0000","last_peered":"2023-12-15T17:23:51.648118+0000","last_clean":"2023-12-15T17:23:51.648118+0000","last_became_active":"2023-12-15T17:03:29.736171+0000","last_became_peered":"2023-12-15T17:03:29.736171+0000","last_unstale":"2023-12-15T17:23:51.648118+0000","last_undegraded":"2023-12-15T17:23:51.648118+0000","last_fullsized":"2023-12-15T17:23:51.648118+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:42:10.131036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":871,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.811275+0000","last_change":"2023-12-15T17:01:59.949551+0000","last_active":"2023-12-15T17:23:51.811275+0000","last_peered":"2023-12-15T17:23:51.811275+0000","last_clean":"2023-12-15T17:23:51.811275+0000","last_became_active":"2023-12-15T17:01:59.918026+0000","last_became_peered":"2023-12-15T17:01:59.918026+0000","last_unstale":"2023-12-15T17:23:51.811275+0000","last_undegraded":"2023-12-15T17:23:51.811275+0000","last_fullsized":"2023-12-15T17:23:51.811275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:10:51.112278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":845,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.834372+0000","last_change":"2023-12-15T17:03:26.256693+0000","last_active":"2023-12-15T17:23:51.834372+0000","last_peered":"2023-12-15T17:23:51.834372+0000","last_clean":"2023-12-15T17:23:51.834372+0000","last_became_active":"2023-12-15T17:03:26.235756+0000","last_became_peered":"2023-12-15T17:03:26.235756+0000","last_unstale":"2023-12-15T17:23:51.834372+0000","last_undegraded":"2023-12-15T17:23:51.834372+0000","last_fullsized":"2023-12-15T17:23:51.834372+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:08:19.767760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":882,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.147791+0000","last_change":"2023-12-15T17:02:01.380902+0000","last_active":"2023-12-15T17:23:56.147791+0000","last_peered":"2023-12-15T17:23:56.147791+0000","last_clean":"2023-12-15T17:23:56.147791+0000","last_became_active":"2023-12-15T17:02:01.375672+0000","last_became_peered":"2023-12-15T17:02:01.375672+0000","last_unstale":"2023-12-15T17:23:56.147791+0000","last_undegraded":"2023-12-15T17:23:56.147791+0000","last_fullsized":"2023-12-15T17:23:56.147791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:15:17.260715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"32'7","reported_seq":863,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.747693+0000","last_change":"2023-12-15T17:03:26.359500+0000","last_active":"2023-12-15T17:23:56.747693+0000","last_peered":"2023-12-15T17:23:56.747693+0000","last_clean":"2023-12-15T17:23:56.747693+0000","last_became_active":"2023-12-15T17:03:26.343248+0000","last_became_peered":"2023-12-15T17:03:26.343248+0000","last_unstale":"2023-12-15T17:23:56.747693+0000","last_undegraded":"2023-12-15T17:23:56.747693+0000","last_fullsized":"2023-12-15T17:23:56.747693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:31:17.788444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":875,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.676345+0000","last_change":"2023-12-15T17:01:59.938931+0000","last_active":"2023-12-15T17:23:51.676345+0000","last_peered":"2023-12-15T17:23:51.676345+0000","last_clean":"2023-12-15T17:23:51.676345+0000","last_became_active":"2023-12-15T17:01:59.908680+0000","last_became_peered":"2023-12-15T17:01:59.908680+0000","last_unstale":"2023-12-15T17:23:51.676345+0000","last_undegraded":"2023-12-15T17:23:51.676345+0000","last_fullsized":"2023-12-15T17:23:51.676345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:35:37.366491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.170131+0000","last_change":"2023-12-15T17:03:27.788302+0000","last_active":"2023-12-15T17:23:56.170131+0000","last_peered":"2023-12-15T17:23:56.170131+0000","last_clean":"2023-12-15T17:23:56.170131+0000","last_became_active":"2023-12-15T17:03:27.780434+0000","last_became_peered":"2023-12-15T17:03:27.780434+0000","last_unstale":"2023-12-15T17:23:56.170131+0000","last_undegraded":"2023-12-15T17:23:56.170131+0000","last_fullsized":"2023-12-15T17:23:56.170131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:24:22.326668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":883,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.126527+0000","last_change":"2023-12-15T17:02:01.718405+0000","last_active":"2023-12-15T17:23:56.126527+0000","last_peered":"2023-12-15T17:23:56.126527+0000","last_clean":"2023-12-15T17:23:56.126527+0000","last_became_active":"2023-12-15T17:02:01.713188+0000","last_became_peered":"2023-12-15T17:02:01.713188+0000","last_unstale":"2023-12-15T17:23:56.126527+0000","last_undegraded":"2023-12-15T17:23:56.126527+0000","last_fullsized":"2023-12-15T17:23:56.126527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:27:19.133287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":865,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.801822+0000","last_change":"2023-12-15T17:03:26.257800+0000","last_active":"2023-12-15T17:23:51.801822+0000","last_peered":"2023-12-15T17:23:51.801822+0000","last_clean":"2023-12-15T17:23:51.801822+0000","last_became_active":"2023-12-15T17:03:26.252650+0000","last_became_peered":"2023-12-15T17:03:26.252650+0000","last_unstale":"2023-12-15T17:23:51.801822+0000","last_undegraded":"2023-12-15T17:23:51.801822+0000","last_fullsized":"2023-12-15T17:23:51.801822+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:20:58.993074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":859,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.780879+0000","last_change":"2023-12-15T17:01:59.559261+0000","last_active":"2023-12-15T17:23:56.780879+0000","last_peered":"2023-12-15T17:23:56.780879+0000","last_clean":"2023-12-15T17:23:56.780879+0000","last_became_active":"2023-12-15T17:01:59.531480+0000","last_became_peered":"2023-12-15T17:01:59.531480+0000","last_unstale":"2023-12-15T17:23:56.780879+0000","last_undegraded":"2023-12-15T17:23:56.780879+0000","last_fullsized":"2023-12-15T17:23:56.780879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:51:07.527384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.146161+0000","last_change":"2023-12-15T17:03:29.375364+0000","last_active":"2023-12-15T17:23:56.146161+0000","last_peered":"2023-12-15T17:23:56.146161+0000","last_clean":"2023-12-15T17:23:56.146161+0000","last_became_active":"2023-12-15T17:03:29.361304+0000","last_became_peered":"2023-12-15T17:03:29.361304+0000","last_unstale":"2023-12-15T17:23:56.146161+0000","last_undegraded":"2023-12-15T17:23:56.146161+0000","last_fullsized":"2023-12-15T17:23:56.146161+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:51:08.058570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":882,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.102856+0000","last_change":"2023-12-15T17:02:01.240306+0000","last_active":"2023-12-15T17:23:56.102856+0000","last_peered":"2023-12-15T17:23:56.102856+0000","last_clean":"2023-12-15T17:23:56.102856+0000","last_became_active":"2023-12-15T17:02:01.208353+0000","last_became_peered":"2023-12-15T17:02:01.208353+0000","last_unstale":"2023-12-15T17:23:56.102856+0000","last_undegraded":"2023-12-15T17:23:56.102856+0000","last_fullsized":"2023-12-15T17:23:56.102856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:41:20.461821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":868,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.659274+0000","last_change":"2023-12-15T17:03:27.328921+0000","last_active":"2023-12-15T17:23:51.659274+0000","last_peered":"2023-12-15T17:23:51.659274+0000","last_clean":"2023-12-15T17:23:51.659274+0000","last_became_active":"2023-12-15T17:03:27.321338+0000","last_became_peered":"2023-12-15T17:03:27.321338+0000","last_unstale":"2023-12-15T17:23:51.659274+0000","last_undegraded":"2023-12-15T17:23:51.659274+0000","last_fullsized":"2023-12-15T17:23:51.659274+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:51:42.546700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":883,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.120609+0000","last_change":"2023-12-15T17:02:01.625426+0000","last_active":"2023-12-15T17:23:56.120609+0000","last_peered":"2023-12-15T17:23:56.120609+0000","last_clean":"2023-12-15T17:23:56.120609+0000","last_became_active":"2023-12-15T17:02:01.616517+0000","last_became_peered":"2023-12-15T17:02:01.616517+0000","last_unstale":"2023-12-15T17:23:56.120609+0000","last_undegraded":"2023-12-15T17:23:56.120609+0000","last_fullsized":"2023-12-15T17:23:56.120609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:54:17.323141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":957,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.831553+0000","last_change":"2023-12-15T17:01:52.907228+0000","last_active":"2023-12-15T17:23:51.831553+0000","last_peered":"2023-12-15T17:23:51.831553+0000","last_clean":"2023-12-15T17:23:51.831553+0000","last_became_active":"2023-12-15T17:01:52.878062+0000","last_became_peered":"2023-12-15T17:01:52.878062+0000","last_unstale":"2023-12-15T17:23:51.831553+0000","last_undegraded":"2023-12-15T17:23:51.831553+0000","last_fullsized":"2023-12-15T17:23:51.831553+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_clean_scrub_stamp":"2023-12-15T17:01:50.252516+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:47:14.717362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":850,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.753308+0000","last_change":"2023-12-15T17:03:31.083758+0000","last_active":"2023-12-15T17:23:56.753308+0000","last_peered":"2023-12-15T17:23:56.753308+0000","last_clean":"2023-12-15T17:23:56.753308+0000","last_became_active":"2023-12-15T17:03:31.075388+0000","last_became_peered":"2023-12-15T17:03:31.075388+0000","last_unstale":"2023-12-15T17:23:56.753308+0000","last_undegraded":"2023-12-15T17:23:56.753308+0000","last_fullsized":"2023-12-15T17:23:56.753308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:32:49.564294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":844,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.858699+0000","last_change":"2023-12-15T17:03:29.701214+0000","last_active":"2023-12-15T17:23:51.858699+0000","last_peered":"2023-12-15T17:23:51.858699+0000","last_clean":"2023-12-15T17:23:51.858699+0000","last_became_active":"2023-12-15T17:03:29.683464+0000","last_became_peered":"2023-12-15T17:03:29.683464+0000","last_unstale":"2023-12-15T17:23:51.858699+0000","last_undegraded":"2023-12-15T17:23:51.858699+0000","last_fullsized":"2023-12-15T17:23:51.858699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:02:06.165049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":853,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.906574+0000","last_change":"2023-12-15T17:03:26.043197+0000","last_active":"2023-12-15T17:23:56.906574+0000","last_peered":"2023-12-15T17:23:56.906574+0000","last_clean":"2023-12-15T17:23:56.906574+0000","last_became_active":"2023-12-15T17:03:26.036270+0000","last_became_peered":"2023-12-15T17:03:26.036270+0000","last_unstale":"2023-12-15T17:23:56.906574+0000","last_undegraded":"2023-12-15T17:23:56.906574+0000","last_fullsized":"2023-12-15T17:23:56.906574+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:27:28.632284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":851,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.960741+0000","last_change":"2023-12-15T17:03:29.615007+0000","last_active":"2023-12-15T17:23:56.960741+0000","last_peered":"2023-12-15T17:23:56.960741+0000","last_clean":"2023-12-15T17:23:56.960741+0000","last_became_active":"2023-12-15T17:03:29.608805+0000","last_became_peered":"2023-12-15T17:03:29.608805+0000","last_unstale":"2023-12-15T17:23:56.960741+0000","last_undegraded":"2023-12-15T17:23:56.960741+0000","last_fullsized":"2023-12-15T17:23:56.960741+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T20:35:50.230477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":869,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.643038+0000","last_change":"2023-12-15T17:03:27.507868+0000","last_active":"2023-12-15T17:23:51.643038+0000","last_peered":"2023-12-15T17:23:51.643038+0000","last_clean":"2023-12-15T17:23:51.643038+0000","last_became_active":"2023-12-15T17:03:27.499200+0000","last_became_peered":"2023-12-15T17:03:27.499200+0000","last_unstale":"2023-12-15T17:23:51.643038+0000","last_undegraded":"2023-12-15T17:23:51.643038+0000","last_fullsized":"2023-12-15T17:23:51.643038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T20:31:29.233160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":869,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.656321+0000","last_change":"2023-12-15T17:03:28.022217+0000","last_active":"2023-12-15T17:23:51.656321+0000","last_peered":"2023-12-15T17:23:51.656321+0000","last_clean":"2023-12-15T17:23:51.656321+0000","last_became_active":"2023-12-15T17:03:28.012736+0000","last_became_peered":"2023-12-15T17:03:28.012736+0000","last_unstale":"2023-12-15T17:23:51.656321+0000","last_undegraded":"2023-12-15T17:23:51.656321+0000","last_fullsized":"2023-12-15T17:23:51.656321+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T23:37:05.516314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":865,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.785043+0000","last_change":"2023-12-15T17:03:27.802217+0000","last_active":"2023-12-15T17:23:51.785043+0000","last_peered":"2023-12-15T17:23:51.785043+0000","last_clean":"2023-12-15T17:23:51.785043+0000","last_became_active":"2023-12-15T17:03:27.791224+0000","last_became_peered":"2023-12-15T17:03:27.791224+0000","last_unstale":"2023-12-15T17:23:51.785043+0000","last_undegraded":"2023-12-15T17:23:51.785043+0000","last_fullsized":"2023-12-15T17:23:51.785043+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:54:03.758840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":845,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.864195+0000","last_change":"2023-12-15T17:03:34.018076+0000","last_active":"2023-12-15T17:23:51.864195+0000","last_peered":"2023-12-15T17:23:51.864195+0000","last_clean":"2023-12-15T17:23:51.864195+0000","last_became_active":"2023-12-15T17:03:33.897262+0000","last_became_peered":"2023-12-15T17:03:33.897262+0000","last_unstale":"2023-12-15T17:23:51.864195+0000","last_undegraded":"2023-12-15T17:23:51.864195+0000","last_fullsized":"2023-12-15T17:23:51.864195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T23:33:26.034906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":868,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.674761+0000","last_change":"2023-12-15T17:03:28.235706+0000","last_active":"2023-12-15T17:23:51.674761+0000","last_peered":"2023-12-15T17:23:51.674761+0000","last_clean":"2023-12-15T17:23:51.674761+0000","last_became_active":"2023-12-15T17:03:28.164968+0000","last_became_peered":"2023-12-15T17:03:28.164968+0000","last_unstale":"2023-12-15T17:23:51.674761+0000","last_undegraded":"2023-12-15T17:23:51.674761+0000","last_fullsized":"2023-12-15T17:23:51.674761+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:55:52.108198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":875,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.143397+0000","last_change":"2023-12-15T17:03:29.256120+0000","last_active":"2023-12-15T17:23:56.143397+0000","last_peered":"2023-12-15T17:23:56.143397+0000","last_clean":"2023-12-15T17:23:56.143397+0000","last_became_active":"2023-12-15T17:03:29.219241+0000","last_became_peered":"2023-12-15T17:03:29.219241+0000","last_unstale":"2023-12-15T17:23:56.143397+0000","last_undegraded":"2023-12-15T17:23:56.143397+0000","last_fullsized":"2023-12-15T17:23:56.143397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:27:29.782496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":843,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.874186+0000","last_change":"2023-12-15T17:03:25.431259+0000","last_active":"2023-12-15T17:23:51.874186+0000","last_peered":"2023-12-15T17:23:51.874186+0000","last_clean":"2023-12-15T17:23:51.874186+0000","last_became_active":"2023-12-15T17:03:25.394333+0000","last_became_peered":"2023-12-15T17:03:25.394333+0000","last_unstale":"2023-12-15T17:23:51.874186+0000","last_undegraded":"2023-12-15T17:23:51.874186+0000","last_fullsized":"2023-12-15T17:23:51.874186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:35:32.135097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":843,"reported_epoch":901,"state":"active+clean","last_fresh":"2023-12-15T17:23:51.904571+0000","last_change":"2023-12-15T17:03:25.933659+0000","last_active":"2023-12-15T17:23:51.904571+0000","last_peered":"2023-12-15T17:23:51.904571+0000","last_clean":"2023-12-15T17:23:51.904571+0000","last_became_active":"2023-12-15T17:03:25.924138+0000","last_became_peered":"2023-12-15T17:03:25.924138+0000","last_unstale":"2023-12-15T17:23:51.904571+0000","last_undegraded":"2023-12-15T17:23:51.904571+0000","last_fullsized":"2023-12-15T17:23:51.904571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:30:47.546645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509707,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58908,"kb_used_data":5512,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312932,"statfs":{"total":96636764160,"available":96576442368,"internally_reserved":0,"allocated":5644288,"data_stored":2992122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":10.289999999999999,"5min":13.026999999999999,"15min":16.065000000000001},"min":{"1min":5.0090000000000003,"5min":4.351,"15min":4.3390000000000004},"max":{"1min":20.521000000000001,"5min":151.61099999999999,"15min":199.32400000000001},"last":17.937000000000001},{"interface":"front","average":{"1min":10.064,"5min":14.347,"15min":17.850999999999999},"min":{"1min":6.2839999999999998,"5min":5.0149999999999997,"15min":4.9370000000000003},"max":{"1min":20.202999999999999,"5min":278.32799999999997,"15min":330.613},"last":21.178000000000001}]},{"osd":1,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":16.135000000000002,"5min":17.571999999999999,"15min":19.690000000000001},"min":{"1min":7.4400000000000004,"5min":6.7439999999999998,"15min":5.726},"max":{"1min":65.194999999999993,"5min":159.22399999999999,"15min":303.95400000000001},"last":7.335},{"interface":"front","average":{"1min":15.039999999999999,"5min":19.803000000000001,"15min":20.053999999999998},"min":{"1min":7.5439999999999996,"5min":6.923,"15min":6.4610000000000003},"max":{"1min":55.744,"5min":326.125,"15min":326.125},"last":21.805}]},{"osd":3,"last update":"Fri Dec 15 17:23:02 2023","interfaces":[{"interface":"back","average":{"1min":10.766999999999999,"5min":17.425999999999998,"15min":23.780000000000001},"min":{"1min":6.6740000000000004,"5min":5.944,"15min":5.9050000000000002},"max":{"1min":30.068000000000001,"5min":309.05399999999997,"15min":313.50400000000002},"last":7.7539999999999996},{"interface":"front","average":{"1min":15.036,"5min":20.998999999999999,"15min":25.573},"min":{"1min":7.476,"5min":7.2480000000000002,"15min":6.1449999999999996},"max":{"1min":28.457999999999998,"5min":372.85199999999998,"15min":372.85199999999998},"last":20.539999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":15.255000000000001,"5min":23.908000000000001,"15min":24.216000000000001},"min":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":6.5359999999999996},"max":{"1min":26.652000000000001,"5min":397.04000000000002,"15min":397.04000000000002},"last":28.001000000000001},{"interface":"front","average":{"1min":16.533999999999999,"5min":24.268999999999998,"15min":25.311},"min":{"1min":8.5980000000000008,"5min":8.5980000000000008,"15min":7.7809999999999997},"max":{"1min":31.385999999999999,"5min":217.01599999999999,"15min":217.01599999999999},"last":16.015000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:08 2023","interfaces":[{"interface":"back","average":{"1min":17.076000000000001,"5min":22.579000000000001,"15min":34.334000000000003},"min":{"1min":8.2629999999999999,"5min":8.2629999999999999,"15min":7.7839999999999998},"max":{"1min":29.724,"5min":196.75700000000001,"15min":493.322},"last":17.254000000000001},{"interface":"front","average":{"1min":13.827,"5min":18.082000000000001,"15min":30.268000000000001},"min":{"1min":7.5880000000000001,"5min":5.9160000000000004,"15min":5.9160000000000004},"max":{"1min":25.835000000000001,"5min":280.45400000000001,"15min":497.67200000000003},"last":22.481999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:49 2023","interfaces":[{"interface":"back","average":{"1min":16.161000000000001,"5min":22.856000000000002,"15min":25.489999999999998},"min":{"1min":10.462,"5min":9.4039999999999999,"15min":8.5969999999999995},"max":{"1min":25.347999999999999,"5min":289.68400000000003,"15min":289.68400000000003},"last":36.061},{"interface":"front","average":{"1min":14.811999999999999,"5min":21.321999999999999,"15min":22.838999999999999},"min":{"1min":9.6989999999999998,"5min":7.8200000000000003,"15min":6.9409999999999998},"max":{"1min":22.036000000000001,"5min":274.80099999999999,"15min":274.80099999999999},"last":26.370000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:02 2023","interfaces":[{"interface":"back","average":{"1min":19.841999999999999,"5min":24.600999999999999,"15min":31.303000000000001},"min":{"1min":9.3930000000000007,"5min":9.3930000000000007,"15min":8.5820000000000007},"max":{"1min":32.054000000000002,"5min":278.98000000000002,"15min":613.21299999999997},"last":29.344999999999999},{"interface":"front","average":{"1min":17.178999999999998,"5min":22.213999999999999,"15min":29.68},"min":{"1min":7.8360000000000003,"5min":7.8360000000000003,"15min":7.8360000000000003},"max":{"1min":29.925999999999998,"5min":275.72300000000001,"15min":613.91099999999994},"last":23.212}]}]},{"osd":1,"up_from":15,"seq":64424509709,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81468,"kb_used_data":5872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290372,"statfs":{"total":96636764160,"available":96553340928,"internally_reserved":0,"allocated":6012928,"data_stored":3389479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":12.414999999999999,"5min":16.533000000000001,"15min":22.98},"min":{"1min":5.2759999999999998,"5min":4.4870000000000001,"15min":4.383},"max":{"1min":36.951999999999998,"5min":94.881,"15min":452.79500000000002},"last":34.609999999999999},{"interface":"front","average":{"1min":16.593,"5min":15.000999999999999,"15min":17.280000000000001},"min":{"1min":5.2960000000000003,"5min":5.0419999999999998,"15min":4.3449999999999998},"max":{"1min":90.009,"5min":127.498,"15min":196.59200000000001},"last":39.939}]},{"osd":2,"last update":"Fri Dec 15 17:23:03 2023","interfaces":[{"interface":"back","average":{"1min":24.509,"5min":19.314,"15min":25.472000000000001},"min":{"1min":6.4480000000000004,"5min":5.7380000000000004,"15min":5.5190000000000001},"max":{"1min":276.47000000000003,"5min":276.47000000000003,"15min":482.72800000000001},"last":95.677999999999997},{"interface":"front","average":{"1min":25.614000000000001,"5min":22.815000000000001,"15min":31.640999999999998},"min":{"1min":8.2119999999999997,"5min":6.5629999999999997,"15min":6.1699999999999999},"max":{"1min":261.74400000000003,"5min":261.74400000000003,"15min":454.017},"last":35.276000000000003}]},{"osd":3,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":19.303000000000001,"5min":20.509,"15min":28.212},"min":{"1min":7.1100000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"max":{"1min":110.723,"5min":147.37100000000001,"15min":488.23700000000002},"last":62.531999999999996},{"interface":"front","average":{"1min":16.472000000000001,"5min":21.302,"15min":26.646000000000001},"min":{"1min":10.347,"5min":9.1549999999999994,"15min":6.4119999999999999},"max":{"1min":30.919,"5min":110.056,"15min":333.53199999999998},"last":61.780999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":13.672000000000001,"5min":18.148,"15min":26.068999999999999},"min":{"1min":7.1959999999999997,"5min":6.2629999999999999,"15min":5.8319999999999999},"max":{"1min":53.085999999999999,"5min":143.24000000000001,"15min":438.75099999999998},"last":93.031999999999996},{"interface":"front","average":{"1min":18.068999999999999,"5min":21.373000000000001,"15min":31.120000000000001},"min":{"1min":10.132999999999999,"5min":8.4510000000000005,"15min":8.4339999999999993},"max":{"1min":41.640000000000001,"5min":78.664000000000001,"15min":443.58300000000003},"last":84.352000000000004}]},{"osd":5,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":22.023,"5min":20.832000000000001,"15min":36.176000000000002},"min":{"1min":8.4710000000000001,"5min":6.8369999999999997,"15min":6.8369999999999997},"max":{"1min":112.105,"5min":113.04300000000001,"15min":451.70999999999998},"last":73.421999999999997},{"interface":"front","average":{"1min":31.739999999999998,"5min":27.744,"15min":29.515999999999998},"min":{"1min":12.116,"5min":9.8840000000000003,"15min":6.6500000000000004},"max":{"1min":209.96600000000001,"5min":209.96600000000001,"15min":264.46899999999999},"last":78.525999999999996}]},{"osd":6,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":21.939,"5min":22.827999999999999,"15min":31.187999999999999},"min":{"1min":12.606,"5min":8.6809999999999992,"15min":7.5629999999999997},"max":{"1min":90.926000000000002,"5min":162.06200000000001,"15min":455.43000000000001},"last":77.608000000000004},{"interface":"front","average":{"1min":19.638999999999999,"5min":21.096,"15min":28.334},"min":{"1min":9.1400000000000006,"5min":7.3399999999999999,"15min":6.7249999999999996},"max":{"1min":94.715999999999994,"5min":169.34,"15min":314.589},"last":146.28399999999999}]},{"osd":7,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":21.369,"5min":26.395,"15min":36.750999999999998},"min":{"1min":10.898999999999999,"5min":10.691000000000001,"15min":10.561999999999999},"max":{"1min":59.499000000000002,"5min":132.423,"15min":453.04300000000001},"last":36.244},{"interface":"front","average":{"1min":20.079999999999998,"5min":21.163,"15min":28.721},"min":{"1min":10.175000000000001,"5min":10.175000000000001,"15min":8.6110000000000007},"max":{"1min":63.219999999999999,"5min":77.674000000000007,"15min":239.51499999999999},"last":45.905000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58892,"kb_used_data":5496,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312948,"statfs":{"total":96636764160,"available":96576458752,"internally_reserved":0,"allocated":5627904,"data_stored":2985572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":13.765000000000001,"5min":13.314,"15min":18.777000000000001},"min":{"1min":6.2359999999999998,"5min":4.3639999999999999,"15min":4.1429999999999998},"max":{"1min":80.552999999999997,"5min":80.552999999999997,"15min":375.77800000000002},"last":10.553000000000001},{"interface":"front","average":{"1min":17.832999999999998,"5min":17.524999999999999,"15min":23.969000000000001},"min":{"1min":6.0199999999999996,"5min":4.8170000000000002,"15min":4.8170000000000002},"max":{"1min":94.995999999999995,"5min":94.995999999999995,"15min":556.42700000000002},"last":7.6689999999999996}]},{"osd":2,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":20.041,"5min":19.704999999999998,"15min":21.385999999999999},"min":{"1min":9.3889999999999993,"5min":6.2939999999999996,"15min":6.2939999999999996},"max":{"1min":99.216999999999999,"5min":248.864,"15min":396.13400000000001},"last":9.2669999999999995},{"interface":"front","average":{"1min":16.388999999999999,"5min":21.504999999999999,"15min":25.138999999999999},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":5.9790000000000001},"max":{"1min":75.944000000000003,"5min":234.161,"15min":354.649},"last":11.680999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":19.158000000000001,"5min":21.491,"15min":22.245000000000001},"min":{"1min":7.6159999999999997,"5min":6.1740000000000004,"15min":6.0830000000000002},"max":{"1min":96.188000000000002,"5min":99.453000000000003,"15min":413.00599999999997},"last":93.516000000000005},{"interface":"front","average":{"1min":15.550000000000001,"5min":19.518999999999998,"15min":24.094999999999999},"min":{"1min":6.0640000000000001,"5min":6.04,"15min":6.04},"max":{"1min":101.57299999999999,"5min":106.357,"15min":526.98599999999999},"last":84.674000000000007}]},{"osd":4,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":21.193999999999999,"5min":24.096,"15min":24.780999999999999},"min":{"1min":7.1829999999999998,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":119.754,"5min":266.279,"15min":389.75700000000001},"last":7.1539999999999999},{"interface":"front","average":{"1min":17.59,"5min":22.911999999999999,"15min":25.393000000000001},"min":{"1min":8.4700000000000006,"5min":8.4700000000000006,"15min":7.3520000000000003},"max":{"1min":37.862000000000002,"5min":227.19200000000001,"15min":436.572},"last":12.255000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":17.655000000000001,"5min":22.420000000000002,"15min":26.187999999999999},"min":{"1min":7.8220000000000001,"5min":7.5359999999999996,"15min":7.3540000000000001},"max":{"1min":67.956999999999994,"5min":121.137,"15min":449.81299999999999},"last":224.11099999999999},{"interface":"front","average":{"1min":20.068000000000001,"5min":23.791,"15min":27.331},"min":{"1min":10.81,"5min":7.7439999999999998,"15min":6.2539999999999996},"max":{"1min":65.381,"5min":109.90000000000001,"15min":429.75299999999999},"last":227.60900000000001}]},{"osd":6,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":19.919,"5min":23.132999999999999,"15min":25.440999999999999},"min":{"1min":10.682,"5min":8.6760000000000002,"15min":7.4699999999999998},"max":{"1min":91.619,"5min":93.299000000000007,"15min":469.86799999999999},"last":15.897},{"interface":"front","average":{"1min":23.565000000000001,"5min":24.164000000000001,"15min":27.797999999999998},"min":{"1min":12.039,"5min":9.2899999999999991,"15min":8.4740000000000002},"max":{"1min":102.664,"5min":102.664,"15min":373.93599999999998},"last":13.752000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":17.504999999999999,"5min":21.597999999999999,"15min":29.437999999999999},"min":{"1min":8.3109999999999999,"5min":7.0449999999999999,"15min":7.0449999999999999},"max":{"1min":77.521000000000001,"5min":151.999,"15min":416.24400000000003},"last":9.9390000000000001},{"interface":"front","average":{"1min":24.347000000000001,"5min":26.122,"15min":27.091000000000001},"min":{"1min":11.058,"5min":9.8610000000000007,"15min":7.7160000000000002},"max":{"1min":100.611,"5min":163.09700000000001,"15min":410.04300000000001},"last":14.868}]}]},{"osd":3,"up_from":15,"seq":64424509706,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76944,"kb_used_data":5508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294896,"statfs":{"total":96636764160,"available":96557973504,"internally_reserved":0,"allocated":5640192,"data_stored":2992117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":8.1609999999999996,"5min":13.029,"15min":18.565999999999999},"min":{"1min":5.3899999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"max":{"1min":14.663,"5min":132.791,"15min":252.483},"last":4.7389999999999999},{"interface":"front","average":{"1min":8.7210000000000001,"5min":13.803000000000001,"15min":18.956},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":24.484999999999999,"5min":138.03800000000001,"15min":235.10300000000001},"last":7.641}]},{"osd":1,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":10.631,"5min":13.651,"15min":22},"min":{"1min":7.4589999999999996,"5min":4.9390000000000001,"15min":4.9390000000000001},"max":{"1min":19.66,"5min":79.147000000000006,"15min":624.94100000000003},"last":21.771999999999998},{"interface":"front","average":{"1min":14.163,"5min":17.541,"15min":24.814},"min":{"1min":8.8680000000000003,"5min":7.4770000000000003,"15min":6.9740000000000002},"max":{"1min":22.484000000000002,"5min":86.152000000000001,"15min":635.13199999999995},"last":32.859999999999999}]},{"osd":2,"last update":"Fri Dec 15 17:23:45 2023","interfaces":[{"interface":"back","average":{"1min":14.015000000000001,"5min":17.581,"15min":22.074999999999999},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":60.210000000000001,"5min":122.60899999999999,"15min":286.22699999999998},"last":10.489000000000001},{"interface":"front","average":{"1min":13.85,"5min":17.390000000000001,"15min":25.114000000000001},"min":{"1min":6.931,"5min":6.931,"15min":6.5380000000000003},"max":{"1min":73.950000000000003,"5min":238.096,"15min":573.05700000000002},"last":14.992000000000001}]},{"osd":4,"last update":"Fri Dec 15 17:23:00 2023","interfaces":[{"interface":"back","average":{"1min":16.614000000000001,"5min":19.768999999999998,"15min":28.959},"min":{"1min":7.5019999999999998,"5min":7.5019999999999998,"15min":7.1159999999999997},"max":{"1min":60.703000000000003,"5min":228.708,"15min":628.97299999999996},"last":9.4979999999999993},{"interface":"front","average":{"1min":19.669,"5min":24.192,"15min":34.237000000000002},"min":{"1min":13.58,"5min":11.423,"15min":8.8239999999999998},"max":{"1min":29.742999999999999,"5min":230.202,"15min":637.40200000000004},"last":18.204999999999998}]},{"osd":5,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":14.532,"5min":20.297999999999998,"15min":26.827999999999999},"min":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.782},"max":{"1min":32.411000000000001,"5min":144.59200000000001,"15min":752.79499999999996},"last":8.6159999999999997},{"interface":"front","average":{"1min":16.890999999999998,"5min":21.039000000000001,"15min":25.788},"min":{"1min":9.5,"5min":9.2129999999999992,"15min":6.0119999999999996},"max":{"1min":45.607999999999997,"5min":145.405,"15min":263.79500000000002},"last":15.510999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":16.896000000000001,"5min":21.753,"15min":27.977},"min":{"1min":9.3599999999999994,"5min":8.0150000000000006,"15min":7.5819999999999999},"max":{"1min":40.414999999999999,"5min":134.255,"15min":498.464},"last":11.379},{"interface":"front","average":{"1min":21.16,"5min":24.161000000000001,"15min":31.126999999999999},"min":{"1min":12.519,"5min":10.362,"15min":8.6560000000000006},"max":{"1min":60.975000000000001,"5min":153.113,"15min":636.16399999999999},"last":20.210000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":17.704999999999998,"5min":23.234000000000002,"15min":31.463000000000001},"min":{"1min":11.615,"5min":10.316000000000001,"15min":8.4930000000000003},"max":{"1min":25.109000000000002,"5min":202.89500000000001,"15min":572.08699999999999},"last":16.465},{"interface":"front","average":{"1min":19.895,"5min":24.616,"15min":31.809999999999999},"min":{"1min":13.664999999999999,"5min":13.173999999999999,"15min":10.406000000000001},"max":{"1min":32.581000000000003,"5min":156.91900000000001,"15min":624.04100000000005},"last":17.247}]}]},{"osd":7,"up_from":14,"seq":60129542411,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76948,"kb_used_data":5508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294892,"statfs":{"total":96636764160,"available":96557969408,"internally_reserved":0,"allocated":5640192,"data_stored":2992117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":63,"apply_latency_ms":63,"commit_latency_ns":63000000,"apply_latency_ns":63000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":11.852,"5min":12.436999999999999,"15min":16.131},"min":{"1min":5.0750000000000002,"5min":4.3810000000000002,"15min":4.3259999999999996},"max":{"1min":37.542000000000002,"5min":91.055999999999997,"15min":161.38499999999999},"last":21.841000000000001},{"interface":"front","average":{"1min":12.800000000000001,"5min":12.664999999999999,"15min":16.463999999999999},"min":{"1min":6.5910000000000002,"5min":4.8339999999999996,"15min":4.8339999999999996},"max":{"1min":64.691999999999993,"5min":93.209999999999994,"15min":142.673},"last":7.3879999999999999}]},{"osd":1,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":20.167000000000002,"5min":18.283000000000001,"15min":22.882000000000001},"min":{"1min":7.2480000000000002,"5min":6.3550000000000004,"15min":5.7629999999999999},"max":{"1min":74.025999999999996,"5min":143.17599999999999,"15min":275.99299999999999},"last":15.551},{"interface":"front","average":{"1min":15.464,"5min":14.946,"15min":21.763999999999999},"min":{"1min":5.9749999999999996,"5min":5.7960000000000003,"15min":5.2649999999999997},"max":{"1min":81.099999999999994,"5min":88.022999999999996,"15min":342.22000000000003},"last":16.494}]},{"osd":2,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":20.966999999999999,"5min":17.347000000000001,"15min":19.687000000000001},"min":{"1min":8.2509999999999994,"5min":7.383,"15min":6.0670000000000002},"max":{"1min":86.772000000000006,"5min":95.926000000000002,"15min":131.81299999999999},"last":26.233000000000001},{"interface":"front","average":{"1min":16.437000000000001,"5min":14.654999999999999,"15min":19.41},"min":{"1min":6.9470000000000001,"5min":6.266,"15min":6.266},"max":{"1min":84.876000000000005,"5min":84.876000000000005,"15min":117.864},"last":23.754999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":20.248000000000001,"5min":20.637,"15min":24.939},"min":{"1min":6.5330000000000004,"5min":6.5330000000000004,"15min":6.4370000000000003},"max":{"1min":118.39400000000001,"5min":118.39400000000001,"15min":390.91000000000003},"last":19.044},{"interface":"front","average":{"1min":20.558,"5min":21.027999999999999,"15min":26.327999999999999},"min":{"1min":8.1479999999999997,"5min":7.4260000000000002,"15min":5.7320000000000002},"max":{"1min":138.559,"5min":138.559,"15min":244.13},"last":24.443999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:23:51 2023","interfaces":[{"interface":"back","average":{"1min":24.532,"5min":22.248999999999999,"15min":24.129999999999999},"min":{"1min":7.3529999999999998,"5min":7.3529999999999998,"15min":7.3529999999999998},"max":{"1min":144.52099999999999,"5min":144.52099999999999,"15min":245.65600000000001},"last":17.129000000000001},{"interface":"front","average":{"1min":25.827999999999999,"5min":24.245999999999999,"15min":23.055},"min":{"1min":9.4199999999999999,"5min":7.726,"15min":6.9139999999999997},"max":{"1min":143.92099999999999,"5min":189.988,"15min":193.07400000000001},"last":21.242000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":17.966000000000001,"5min":19.504000000000001,"15min":25.530000000000001},"min":{"1min":7.7699999999999996,"5min":7.7699999999999996,"15min":7.7699999999999996},"max":{"1min":69.656999999999996,"5min":109.459,"15min":198.96000000000001},"last":22.948},{"interface":"front","average":{"1min":21.565999999999999,"5min":20.701000000000001,"15min":27.978999999999999},"min":{"1min":8.7959999999999994,"5min":7.6189999999999998,"15min":7.6189999999999998},"max":{"1min":145.94,"5min":165.97999999999999,"15min":266.33699999999999},"last":17.718}]},{"osd":6,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":24.082000000000001,"5min":22.300000000000001,"15min":30.507000000000001},"min":{"1min":11.667999999999999,"5min":7.952,"15min":7.8949999999999996},"max":{"1min":112.369,"5min":113.128,"15min":380.62700000000001},"last":25.324999999999999},{"interface":"front","average":{"1min":23.890999999999998,"5min":21.359000000000002,"15min":26.186},"min":{"1min":8.3550000000000004,"5min":8.3550000000000004,"15min":7.2270000000000003},"max":{"1min":108.03,"5min":108.03,"15min":226.91499999999999},"last":18.34}]}]},{"osd":6,"up_from":14,"seq":60129542411,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77028,"kb_used_data":5592,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294812,"statfs":{"total":96636764160,"available":96557887488,"internally_reserved":0,"allocated":5726208,"data_stored":3106084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":10.004,"5min":12.342000000000001,"15min":18.960000000000001},"min":{"1min":4.4269999999999996,"5min":4.133,"15min":4.133},"max":{"1min":25.085999999999999,"5min":128.82499999999999,"15min":223.15199999999999},"last":122.229},{"interface":"front","average":{"1min":9.6329999999999991,"5min":14.333,"15min":18.896000000000001},"min":{"1min":6.2009999999999996,"5min":5.3490000000000002,"15min":4.6920000000000002},"max":{"1min":17.561,"5min":120.85299999999999,"15min":190.249},"last":106.575}]},{"osd":1,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":12.476000000000001,"5min":18.774000000000001,"15min":22.052},"min":{"1min":7.2279999999999998,"5min":5.1319999999999997,"15min":5.0449999999999999},"max":{"1min":24.771999999999998,"5min":177.15100000000001,"15min":292.72199999999998},"last":128.77000000000001},{"interface":"front","average":{"1min":14.473000000000001,"5min":19.847999999999999,"15min":25.225000000000001},"min":{"1min":7.6120000000000001,"5min":6.4390000000000001,"15min":5.0309999999999997},"max":{"1min":41.270000000000003,"5min":174.02199999999999,"15min":422.36099999999999},"last":61.049999999999997}]},{"osd":2,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":17.154,"5min":17.030000000000001,"15min":19.913},"min":{"1min":7.7169999999999996,"5min":6.2850000000000001,"15min":5.2699999999999996},"max":{"1min":66.974000000000004,"5min":131.71000000000001,"15min":131.863},"last":128.06100000000001},{"interface":"front","average":{"1min":13.039,"5min":19.218,"15min":23.687000000000001},"min":{"1min":6.9530000000000003,"5min":6.9530000000000003,"15min":6.7439999999999998},"max":{"1min":65.632000000000005,"5min":112.548,"15min":189.417},"last":59.667000000000002}]},{"osd":3,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":21.212,"5min":24.562000000000001,"15min":27.657},"min":{"1min":7.7279999999999998,"5min":7.7279999999999998,"15min":7.7279999999999998},"max":{"1min":77.271000000000001,"5min":263.45299999999997,"15min":263.45299999999997},"last":118.636},{"interface":"front","average":{"1min":24.375,"5min":27.704000000000001,"15min":30.245999999999999},"min":{"1min":11.551,"5min":10.195,"15min":6.9100000000000001},"max":{"1min":83.891999999999996,"5min":264.54000000000002,"15min":356.815},"last":120.664}]},{"osd":4,"last update":"Fri Dec 15 17:23:25 2023","interfaces":[{"interface":"back","average":{"1min":17.318000000000001,"5min":20.620000000000001,"15min":24.326000000000001},"min":{"1min":6.4020000000000001,"5min":6.4020000000000001,"15min":6.4020000000000001},"max":{"1min":118.277,"5min":122.881,"15min":224.96299999999999},"last":127.682},{"interface":"front","average":{"1min":16.988,"5min":21.347000000000001,"15min":26.774000000000001},"min":{"1min":6.9100000000000001,"5min":6.9100000000000001,"15min":6.9100000000000001},"max":{"1min":99.513000000000005,"5min":127.607,"15min":266.88400000000001},"last":59.033000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":15.765000000000001,"5min":21.884,"15min":28.344000000000001},"min":{"1min":7.9089999999999998,"5min":7.601,"15min":7.0369999999999999},"max":{"1min":40.454999999999998,"5min":164.62899999999999,"15min":349.983},"last":123.834},{"interface":"front","average":{"1min":14.981999999999999,"5min":21.09,"15min":25.739000000000001},"min":{"1min":9.1750000000000007,"5min":7.4530000000000003,"15min":6.8449999999999998},"max":{"1min":26.895,"5min":115.54900000000001,"15min":324.95299999999997},"last":114.51900000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:47 2023","interfaces":[{"interface":"back","average":{"1min":15.082000000000001,"5min":22.818999999999999,"15min":28.507000000000001},"min":{"1min":9.4640000000000004,"5min":8.2110000000000003,"15min":6.8520000000000003},"max":{"1min":38.795999999999999,"5min":137.59700000000001,"15min":268.66500000000002},"last":71.492999999999995},{"interface":"front","average":{"1min":18.492999999999999,"5min":26.584,"15min":33.052999999999997},"min":{"1min":12.459,"5min":8.8849999999999998,"15min":8.8849999999999998},"max":{"1min":48.68,"5min":135.74199999999999,"15min":318.56900000000002},"last":64.582999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542412,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81484,"kb_used_data":5880,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290356,"statfs":{"total":96636764160,"available":96553324544,"internally_reserved":0,"allocated":6021120,"data_stored":3397428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":66,"apply_latency_ms":66,"commit_latency_ns":66000000,"apply_latency_ns":66000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":10.813000000000001,"5min":18.579999999999998,"15min":19.327999999999999},"min":{"1min":5.0339999999999998,"5min":4.7380000000000004,"15min":4.2859999999999996},"max":{"1min":24.373000000000001,"5min":209.15799999999999,"15min":223.69499999999999},"last":8.3170000000000002},{"interface":"front","average":{"1min":10.352,"5min":17.178999999999998,"15min":19.341999999999999},"min":{"1min":4.2590000000000003,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":17.353000000000002,"5min":183.93899999999999,"15min":226.76300000000001},"last":15.997}]},{"osd":1,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":13.563000000000001,"5min":17.940000000000001,"15min":22.196000000000002},"min":{"1min":6.6539999999999999,"5min":5.1260000000000003,"15min":4.7839999999999998},"max":{"1min":63.030999999999999,"5min":137.07499999999999,"15min":340.32999999999998},"last":10.321999999999999},{"interface":"front","average":{"1min":18.015999999999998,"5min":20.047999999999998,"15min":24.577000000000002},"min":{"1min":8.7149999999999999,"5min":5.7850000000000001,"15min":5.6550000000000002},"max":{"1min":57.795000000000002,"5min":169.648,"15min":326.375},"last":13.653}]},{"osd":2,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":17.568000000000001,"5min":20.483000000000001,"15min":22.577000000000002},"min":{"1min":6.6630000000000003,"5min":6.6630000000000003,"15min":5.3979999999999997},"max":{"1min":92.230000000000004,"5min":174.09100000000001,"15min":240.00999999999999},"last":8.9309999999999992},{"interface":"front","average":{"1min":21.774999999999999,"5min":24.068000000000001,"15min":25.559000000000001},"min":{"1min":6.7690000000000001,"5min":5.569,"15min":5.569},"max":{"1min":128.98099999999999,"5min":228.31899999999999,"15min":371.30099999999999},"last":9.6449999999999996}]},{"osd":3,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":25.259,"5min":25.724,"15min":28.169},"min":{"1min":9.2319999999999993,"5min":6.5999999999999996,"15min":6.5999999999999996},"max":{"1min":100.711,"5min":177.51300000000001,"15min":321.26100000000002},"last":7.2320000000000002},{"interface":"front","average":{"1min":25.388999999999999,"5min":26.262,"15min":27.506},"min":{"1min":7.3799999999999999,"5min":7.101,"15min":6.8730000000000002},"max":{"1min":101.599,"5min":172.887,"15min":284.12599999999998},"last":14.48}]},{"osd":4,"last update":"Fri Dec 15 17:23:40 2023","interfaces":[{"interface":"back","average":{"1min":18.481000000000002,"5min":29.667999999999999,"15min":27.547000000000001},"min":{"1min":8.3559999999999999,"5min":8.3559999999999999,"15min":6.8600000000000003},"max":{"1min":61.365000000000002,"5min":229.077,"15min":376.39800000000002},"last":11.821},{"interface":"front","average":{"1min":19.219000000000001,"5min":28.552,"15min":27.09},"min":{"1min":10.989000000000001,"5min":7.8840000000000003,"15min":6.2789999999999999},"max":{"1min":55.509999999999998,"5min":211.03100000000001,"15min":338.56400000000002},"last":16.658999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":25.266999999999999,"15min":27.039000000000001},"min":{"1min":12.606999999999999,"5min":7.8079999999999998,"15min":7.0789999999999997},"max":{"1min":57.170000000000002,"5min":215.21299999999999,"15min":378.327},"last":15.281000000000001},{"interface":"front","average":{"1min":21.053000000000001,"5min":29.216999999999999,"15min":28.443000000000001},"min":{"1min":9.3049999999999997,"5min":8.8900000000000006,"15min":6.6369999999999996},"max":{"1min":62.055999999999997,"5min":228.58799999999999,"15min":407.16500000000002},"last":11.048999999999999}]},{"osd":7,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":19.960999999999999,"5min":24.295000000000002,"15min":28.638999999999999},"min":{"1min":12.951000000000001,"5min":10.864000000000001,"15min":7.7599999999999998},"max":{"1min":39.515999999999998,"5min":151.386,"15min":292.82299999999998},"last":18.800000000000001},{"interface":"front","average":{"1min":20.140000000000001,"5min":27.221,"15min":30.507000000000001},"min":{"1min":11.416,"5min":9.2170000000000005,"15min":7.8650000000000002},"max":{"1min":28.832999999999998,"5min":190.74600000000001,"15min":312.73599999999999},"last":17.997}]}]},{"osd":4,"up_from":14,"seq":60129542412,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58876,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312964,"statfs":{"total":96636764160,"available":96576475136,"internally_reserved":0,"allocated":5611520,"data_stored":2995730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":20.422000000000001,"5min":13.071,"15min":15.058},"min":{"1min":4.7779999999999996,"5min":4.4560000000000004,"15min":4.1289999999999996},"max":{"1min":62.142000000000003,"5min":62.142000000000003,"15min":193.11500000000001},"last":14.269},{"interface":"front","average":{"1min":27.135999999999999,"5min":13.522,"15min":16.253},"min":{"1min":5.9699999999999998,"5min":4.4109999999999996,"15min":4.3719999999999999},"max":{"1min":96.224000000000004,"5min":96.224000000000004,"15min":235.09800000000001},"last":8.141}]},{"osd":1,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":26.41,"5min":16.433,"15min":19.759},"min":{"1min":8.8640000000000008,"5min":5.2199999999999998,"15min":4.9930000000000003},"max":{"1min":86.936999999999998,"5min":86.936999999999998,"15min":289.11099999999999},"last":17.602},{"interface":"front","average":{"1min":22.891999999999999,"5min":18.584,"15min":21.152999999999999},"min":{"1min":7.3520000000000003,"5min":6.2549999999999999,"15min":6.2249999999999996},"max":{"1min":83.016999999999996,"5min":130.535,"15min":218.32400000000001},"last":9.3420000000000005}]},{"osd":2,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":35.674999999999997,"5min":20.347999999999999,"15min":18.050000000000001},"min":{"1min":8.2569999999999997,"5min":6.8179999999999996,"15min":5.5880000000000001},"max":{"1min":163.31399999999999,"5min":163.31399999999999,"15min":163.31399999999999},"last":10.619},{"interface":"front","average":{"1min":31.010000000000002,"5min":19.277999999999999,"15min":18.506},"min":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":5.4690000000000003},"max":{"1min":134.78700000000001,"5min":134.78700000000001,"15min":140.416},"last":15.731999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":26.495000000000001,"5min":23.353000000000002,"15min":21.949000000000002},"min":{"1min":11.071999999999999,"5min":8.9169999999999998,"15min":7.2450000000000001},"max":{"1min":89.188000000000002,"5min":230.505,"15min":230.505},"last":18.855},{"interface":"front","average":{"1min":22.827000000000002,"5min":19.329999999999998,"15min":21.117999999999999},"min":{"1min":9.0299999999999994,"5min":6.4210000000000003,"15min":6.4210000000000003},"max":{"1min":84.497,"5min":136.66499999999999,"15min":203.99100000000001},"last":12.129}]},{"osd":5,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":19.713000000000001,"5min":19.503,"15min":21.353999999999999},"min":{"1min":7.7530000000000001,"5min":6.9930000000000003,"15min":6.9930000000000003},"max":{"1min":87.989999999999995,"5min":123.71599999999999,"15min":162.02099999999999},"last":22.452999999999999},{"interface":"front","average":{"1min":23.210999999999999,"5min":23.539999999999999,"15min":24.937999999999999},"min":{"1min":10.042999999999999,"5min":10.042999999999999,"15min":7.1130000000000004},"max":{"1min":89.099999999999994,"5min":159.16999999999999,"15min":162.922},"last":23.311}]},{"osd":6,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":25.030000000000001,"5min":19.335999999999999,"15min":22.289999999999999},"min":{"1min":7.5579999999999998,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":84.040000000000006,"5min":85.606999999999999,"15min":744.33199999999999},"last":13.097},{"interface":"front","average":{"1min":38.479999999999997,"5min":26.033000000000001,"15min":25.606000000000002},"min":{"1min":14.532999999999999,"5min":10.544,"15min":9.1799999999999997},"max":{"1min":157.441,"5min":157.441,"15min":251.19200000000001},"last":19.991}]},{"osd":7,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":31.425999999999998,"5min":25.463000000000001,"15min":26.760999999999999},"min":{"1min":17.059000000000001,"5min":11.148999999999999,"15min":9.7330000000000005},"max":{"1min":91.006,"5min":144.72,"15min":202.46600000000001},"last":21.131},{"interface":"front","average":{"1min":21.945,"5min":18.006,"15min":20.908000000000001},"min":{"1min":9.6690000000000005,"5min":7.5860000000000003,"15min":6.5430000000000001},"max":{"1min":78.950000000000003,"5min":85.629999999999995,"15min":199.01400000000001},"last":15.035}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-15T17:23:59.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T17:23:59.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.493+0000 7fed002a2640 10 monclient: get_monmap_and_config 2023-12-15T17:23:59.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.493+0000 7fed002a2640 10 monclient: build_initial_monmap 2023-12-15T17:23:59.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.493+0000 7fed002a2640 10 monclient: monmap: 2023-12-15T17:23:59.494 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:23:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:23:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:23:59.494642+0000 2023-12-15T17:23:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:23:59.494642+0000 2023-12-15T17:23:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:23:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:23:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:23:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:23:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:23:59.495 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:23:59.495 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.493+0000 7fed002a2640 1 Processor -- start 2023-12-15T17:23:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.493+0000 7fed002a2640 1 -- start start 2023-12-15T17:23:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.493+0000 7fed002a2640 10 monclient: init 2023-12-15T17:23:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 10 monclient: _reopen_session rank -1 2023-12-15T17:23:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-12-15T17:23:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fecf80a8a30 0x7fecf81424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 10 monclient(hunting): picked mon.noname-c con 0x7fecf80a8a30 addr v2:172.21.15.192:3300/0 2023-12-15T17:23:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fecf80a8e10 addr v1:172.21.15.69:6789/0 2023-12-15T17:23:59.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fecf8142a20 0x7fecf8145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 10 monclient(hunting): picked mon.noname-a con 0x7fecf8142a20 addr v2:172.21.15.69:3300/0 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf8142e70 con 0x7fecf80a8e10 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf80aaa10 con 0x7fecf8142a20 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf80aac10 con 0x7fecf80a8a30 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 10 monclient(hunting): _renew_subs 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.494+0000 7fed002a2640 10 monclient(hunting): authenticate will time out at 2743.810913s 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe83f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fecf8142a20 0x7fecf8145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe03e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fecf80a8a30 0x7fecf81424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe03e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fecf80a8a30 0x7fecf81424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:43830/0 (socket says 172.21.15.69:43830) 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe03e640 1 -- 172.21.15.69:0/819700728 learned_addr learned my addr 172.21.15.69:0/819700728 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:23:59.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe03e640 10 monclient(hunting): get_auth_request con 0x7fecf80a8a30 auth_method 0 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe03e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe03e640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe03e640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe83f640 10 monclient(hunting): get_auth_request con 0x7fecf8142a20 auth_method 0 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe83f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe83f640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe83f640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe03e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe03e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.495+0000 7fecfe03e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.496+0000 7fecfe83f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.496+0000 7fecfe83f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:23:59.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.496+0000 7fecfe83f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:23:59.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.497+0000 7fecfe03e640 10 monclient(hunting): handle_auth_done global_id 21412 payload 274 2023-12-15T17:23:59.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.497+0000 7fecfe03e640 1 -- 172.21.15.69:0/819700728 >> v2:172.21.15.69:3300/0 conn(0x7fecf8142a20 msgr2=0x7fecf8145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:23:59.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.497+0000 7fecfe03e640 1 --2- 172.21.15.69:0/819700728 >> v2:172.21.15.69:3300/0 conn(0x7fecf8142a20 0x7fecf8145bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.497+0000 7fecfe03e640 1 -- 172.21.15.69:0/819700728 >> v1:172.21.15.69:6789/0 conn(0x7fecf80a8e10 legacy=0x7fecf81454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:23:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.497+0000 7fecfe03e640 10 monclient: _finish_hunting 0 2023-12-15T17:23:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.497+0000 7fecfe03e640 1 monclient: found mon.noname-c 2023-12-15T17:23:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.497+0000 7fecfe03e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:23:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.497+0000 7fecfe03e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:23:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.497+0000 7fecfe03e640 1 -- 172.21.15.69:0/819700728 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf80aaef0 con 0x7fecf80a8a30 2023-12-15T17:23:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.499+0000 7fecfe03e640 1 --2- 172.21.15.69:0/819700728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecf80a8a30 0x7fecf81424e0 secure :-1 s=READY pgs=5099 cs=0 l=1 rev1=1 crypto rx=0x7fecf80b5470 tx=0x7fecf402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5090dd636b2b4280 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:23:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.502+0000 7fecfd03c640 1 -- 172.21.15.69:0/819700728 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4030860 con 0x7fecf80a8a30 2023-12-15T17:23:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.502+0000 7fecfd03c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:23:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.502+0000 7fecfd03c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:23:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.502+0000 7fecfd03c640 10 monclient: dump: 2023-12-15T17:23:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:23:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:23:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:59.504 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fecfd03c640 10 monclient: _finish_auth 0 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fecfd03c640 10 monclient: _check_auth_tickets 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fecfd03c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fecfd03c640 1 -- 172.21.15.69:0/819700728 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf40309c0 con 0x7fecf80a8a30 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 5 monclient: authenticate success, global_id 21412 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fecfd03c640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 10 monclient: get_monmap_and_config success 2023-12-15T17:23:59.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fecfd03c640 1 -- 172.21.15.69:0/819700728 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4030d50 con 0x7fecf80a8a30 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fecfd03c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 4 set_mon_vals no callback set 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fecfd03c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fecfd03c640 10 monclient: dump: 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:23:59.506 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 10 monclient: shutdown 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 1 -- 172.21.15.69:0/819700728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecf80a8a30 msgr2=0x7fecf81424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 1 --2- 172.21.15.69:0/819700728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecf80a8a30 0x7fecf81424e0 secure :-1 s=READY pgs=5099 cs=0 l=1 rev1=1 crypto rx=0x7fecf80b5470 tx=0x7fecf402b9a0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 1 -- 172.21.15.69:0/819700728 shutdown_connections 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 1 --2- 172.21.15.69:0/819700728 >> v2:172.21.15.69:3300/0 conn(0x7fecf8142a20 0x7fecf8145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 1 --2- 172.21.15.69:0/819700728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecf80a8a30 0x7fecf81424e0 unknown :-1 s=CLOSED pgs=5099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 1 -- 172.21.15.69:0/819700728 >> 172.21.15.69:0/819700728 conn(0x7fecf8061a30 msgr2=0x7fecf80b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 1 -- 172.21.15.69:0/819700728 shutdown_connections 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.503+0000 7fed002a2640 1 -- 172.21.15.69:0/819700728 wait complete. 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.504+0000 7fed002a2640 10 monclient: build_initial_monmap 2023-12-15T17:23:59.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.504+0000 7fed002a2640 10 monclient: monmap: 2023-12-15T17:23:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:23:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:23:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:23:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:23:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:23:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:23:59.511 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.504+0000 7fed002a2640 1 librados: starting msgr at 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.504+0000 7fed002a2640 1 librados: starting objecter 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.504+0000 7fed002a2640 1 Processor -- start 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.504+0000 7fed002a2640 1 -- start start 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.504+0000 7fed002a2640 1 librados: setting wanted keys 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.504+0000 7fed002a2640 1 librados: calling monclient init 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.504+0000 7fed002a2640 10 monclient: init 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 10 monclient: _reopen_session rank -1 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecf80a8a30 0x7fecf813baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 10 monclient(hunting): picked mon.noname-b con 0x7fecf80a8a30 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecf80a8e10 0x7fecf81dad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:59.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 10 monclient(hunting): picked mon.noname-c con 0x7fecf80a8e10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf8142a20 0x7fecf81db290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 10 monclient(hunting): picked mon.noname-a con 0x7fecf8142a20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf81db7d0 con 0x7fecf8142a20 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf80b0830 con 0x7fecf80a8a30 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf80af150 con 0x7fecf80a8e10 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 10 monclient(hunting): _renew_subs 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fed002a2640 10 monclient(hunting): authenticate will time out at 2743.821771s 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.505+0000 7fecfe83f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf8142a20 0x7fecf81db290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe03e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecf80a8a30 0x7fecf813baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfd83d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecf80a8e10 0x7fecf81dad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:59.513 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe83f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf8142a20 0x7fecf81db290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45300/0 (socket says 172.21.15.69:45300) 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe83f640 1 -- 172.21.15.69:0/2422125496 learned_addr learned my addr 172.21.15.69:0/2422125496 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe83f640 10 monclient(hunting): get_auth_request con 0x7fecf8142a20 auth_method 0 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe83f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe83f640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe83f640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe03e640 10 monclient(hunting): get_auth_request con 0x7fecf80a8a30 auth_method 0 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe03e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe03e640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe03e640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfd83d640 10 monclient(hunting): get_auth_request con 0x7fecf80a8e10 auth_method 0 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfd83d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfd83d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfd83d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe03e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:23:59.514 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe03e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:23:59.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe03e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:23:59.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe83f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:23:59.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe83f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:23:59.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfe83f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:23:59.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfd83d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:23:59.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.506+0000 7fecfd83d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:23:59.515 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.507+0000 7fecfd83d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:23:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.509+0000 7fecfe03e640 10 monclient(hunting): handle_auth_done global_id 21415 payload 824 2023-12-15T17:23:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.509+0000 7fecfe03e640 1 -- 172.21.15.69:0/2422125496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecf80a8e10 msgr2=0x7fecf81dad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:23:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.509+0000 7fecfe03e640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecf80a8e10 0x7fecf81dad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.509+0000 7fecfe03e640 1 -- 172.21.15.69:0/2422125496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf8142a20 msgr2=0x7fecf81db290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:23:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.509+0000 7fecfe03e640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf8142a20 0x7fecf81db290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.509+0000 7fecfe03e640 10 monclient: _finish_hunting 0 2023-12-15T17:23:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.509+0000 7fecfe03e640 1 monclient: found mon.noname-b 2023-12-15T17:23:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.509+0000 7fecfe03e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:23:59.516 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.509+0000 7fecfe03e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:23:59.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.509+0000 7fecfe03e640 1 -- 172.21.15.69:0/2422125496 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40094d0 con 0x7fecf80a8a30 2023-12-15T17:23:59.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.511+0000 7fecfe03e640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecf80a8a30 0x7fecf813baa0 secure :-1 s=READY pgs=5100 cs=0 l=1 rev1=1 crypto rx=0x7fecf4033000 tx=0x7fecf4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:23:59.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.511+0000 7feceeffd640 10 client.?.objecter ms_handle_connect 0x7fecf80a8a30 2023-12-15T17:23:59.517 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.511+0000 7feceeffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:23:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 1 -- 172.21.15.69:0/2422125496 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4039c90 con 0x7fecf80a8a30 2023-12-15T17:23:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:23:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:23:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 10 monclient: dump: 2023-12-15T17:23:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:23:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:23:59.518 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 10 monclient: _finish_auth 0 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 10 monclient: _check_auth_tickets 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 1 -- 172.21.15.69:0/2422125496 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf4039df0 con 0x7fecf80a8a30 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 1 -- 172.21.15.69:0/2422125496 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4043640 con 0x7fecf80a8a30 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:23:59.519 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7feceeffd640 10 monclient: dump: 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7fed002a2640 5 monclient: authenticate success, global_id 21415 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7fecfc83b640 4 set_mon_vals no callback set 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7fed002a2640 10 monclient: _renew_subs 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7fed002a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:23:59.520 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7fed002a2640 1 -- 172.21.15.69:0/2422125496 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf80af3d0 con 0x7fecf80a8a30 2023-12-15T17:23:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7fed002a2640 10 client.21415.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:23:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7fed002a2640 10 monclient: _renew_subs 2023-12-15T17:23:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7fed002a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:23:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7fed002a2640 1 -- 172.21.15.69:0/2422125496 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf80af740 con 0x7fecf80a8a30 2023-12-15T17:23:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.515+0000 7fed002a2640 1 librados: init done 2023-12-15T17:23:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.516+0000 7fed002a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:23:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.516+0000 7fed002a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:23:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.516+0000 7fed002a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:23:59.521 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.516+0000 7fed002a2640 1 -- 172.21.15.69:0/2422125496 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf8067ac0 con 0x7fecf80a8a30 2023-12-15T17:23:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.534+0000 7feceeffd640 1 -- 172.21.15.69:0/2422125496 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fecf40437a0 con 0x7fecf80a8a30 2023-12-15T17:23:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.534+0000 7feceeffd640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fecdc06cb50 0x7fecdc06f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:23:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.535+0000 7fecfd83d640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fecdc06cb50 0x7fecdc06f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:23:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.535+0000 7fecfd83d640 10 monclient: get_auth_request con 0x7fecdc06cb50 auth_method 0 2023-12-15T17:23:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.535+0000 7feceeffd640 10 client.21415.objecter ms_handle_connect 0x7fecdc06cb50 2023-12-15T17:23:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.535+0000 7fecfd83d640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fecdc06cb50 0x7fecdc06f010 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fecf806c280 tx=0x7fece8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:23:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.759+0000 7feceeffd640 1 -- 172.21.15.69:0/2422125496 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 534..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7fecf403e070 con 0x7fecf80a8a30 2023-12-15T17:23:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.759+0000 7feceeffd640 10 client.21415.objecter ms_dispatch 0x7fecf8000b90 osd_map(908..908 src has 534..908) v4 2023-12-15T17:23:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.759+0000 7feceeffd640 3 client.21415.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:23:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.759+0000 7feceeffd640 3 client.21415.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:23:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.759+0000 7feceeffd640 20 client.21415.objecter dump_active .. 0 homeless 2023-12-15T17:23:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.770+0000 7feceeffd640 1 -- 172.21.15.69:0/2422125496 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fecf4078250 con 0x7fecf80a8a30 2023-12-15T17:23:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.770+0000 7feceeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:23:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.770+0000 7feceeffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:23:59.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.934+0000 7fed002a2640 1 -- 172.21.15.69:0/2422125496 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fecf8142700 con 0x7fecdc06cb50 2023-12-15T17:23:59.940 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.939+0000 7feceeffd640 1 -- 172.21.15.69:0/2422125496 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151952 (secure 0 0 0) 0x7fecf8142700 con 0x7fecdc06cb50 2023-12-15T17:23:59.942 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:23:59.942 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-12-15T17:23:59.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 10 librados: watch_flush enter 2023-12-15T17:23:59.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 10 librados: watch_flush exit 2023-12-15T17:23:59.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 20 client.21415.objecter shutdown clearing up homeless session... 2023-12-15T17:23:59.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 10 client.21415.objecter successfully canceled tick 2023-12-15T17:23:59.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 -- 172.21.15.69:0/2422125496 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fecdc06cb50 msgr2=0x7fecdc06f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:23:59.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fecdc06cb50 0x7fecdc06f010 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fecf806c280 tx=0x7fece8009040 comp rx=0 tx=0).stop 2023-12-15T17:23:59.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 10 monclient: shutdown 2023-12-15T17:23:59.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:23:59.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 -- 172.21.15.69:0/2422125496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecf80a8a30 msgr2=0x7fecf813baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:23:59.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecf80a8a30 0x7fecf813baa0 secure :-1 s=READY pgs=5100 cs=0 l=1 rev1=1 crypto rx=0x7fecf4033000 tx=0x7fecf4004730 comp rx=0 tx=0).stop 2023-12-15T17:23:59.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 -- 172.21.15.69:0/2422125496 shutdown_connections 2023-12-15T17:23:59.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fecdc06cb50 0x7fecdc06f010 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf8142a20 0x7fecf81db290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecf80a8e10 0x7fecf81dad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 --2- 172.21.15.69:0/2422125496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecf80a8a30 0x7fecf813baa0 unknown :-1 s=CLOSED pgs=5100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:23:59.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 -- 172.21.15.69:0/2422125496 >> 172.21.15.69:0/2422125496 conn(0x7fecf8061a30 msgr2=0x7fecf80b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:23:59.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 -- 172.21.15.69:0/2422125496 shutdown_connections 2023-12-15T17:23:59.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 -- 172.21.15.69:0/2422125496 wait complete. 2023-12-15T17:23:59.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:23:59.941+0000 7fed002a2640 1 librados: shutdown 2023-12-15T17:23:59.954 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1579,"stamp":"2023-12-15T17:23:59.782162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":570600,"kb_used_data":44892,"kb_used_omap":12,"kb_used_meta":525555,"kb_avail":754404120,"statfs":{"total":773094113280,"available":772509818880,"internally_reserved":0,"allocated":45969408,"data_stored":24753545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":538168880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":394,"apply_latency_ms":394,"commit_latency_ns":394000000,"apply_latency_ns":394000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000712"},"pg_stats":[{"pgid":"7.1b","version":"0'0","reported_seq":869,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.257675+0000","last_change":"2023-12-15T17:03:26.413418+0000","last_active":"2023-12-15T17:23:57.257675+0000","last_peered":"2023-12-15T17:23:57.257675+0000","last_clean":"2023-12-15T17:23:57.257675+0000","last_became_active":"2023-12-15T17:03:26.401879+0000","last_became_peered":"2023-12-15T17:03:26.401879+0000","last_unstale":"2023-12-15T17:23:57.257675+0000","last_undegraded":"2023-12-15T17:23:57.257675+0000","last_fullsized":"2023-12-15T17:23:57.257675+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:25:02.124791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.128322+0000","last_change":"2023-12-15T17:03:43.168165+0000","last_active":"2023-12-15T17:23:56.128322+0000","last_peered":"2023-12-15T17:23:56.128322+0000","last_clean":"2023-12-15T17:23:56.128322+0000","last_became_active":"2023-12-15T17:03:43.159317+0000","last_became_peered":"2023-12-15T17:03:43.159317+0000","last_unstale":"2023-12-15T17:23:56.128322+0000","last_undegraded":"2023-12-15T17:23:56.128322+0000","last_fullsized":"2023-12-15T17:23:56.128322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:28:01.151029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.123155+0000","last_change":"2023-12-15T17:03:27.146666+0000","last_active":"2023-12-15T17:23:56.123155+0000","last_peered":"2023-12-15T17:23:56.123155+0000","last_clean":"2023-12-15T17:23:56.123155+0000","last_became_active":"2023-12-15T17:03:27.137993+0000","last_became_peered":"2023-12-15T17:03:27.137993+0000","last_unstale":"2023-12-15T17:23:56.123155+0000","last_undegraded":"2023-12-15T17:23:56.123155+0000","last_fullsized":"2023-12-15T17:23:56.123155+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:13:48.832295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":869,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.238398+0000","last_change":"2023-12-15T17:03:32.164840+0000","last_active":"2023-12-15T17:23:57.238398+0000","last_peered":"2023-12-15T17:23:57.238398+0000","last_clean":"2023-12-15T17:23:57.238398+0000","last_became_active":"2023-12-15T17:03:32.155432+0000","last_became_peered":"2023-12-15T17:03:32.155432+0000","last_unstale":"2023-12-15T17:23:57.238398+0000","last_undegraded":"2023-12-15T17:23:57.238398+0000","last_fullsized":"2023-12-15T17:23:57.238398+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:29:00.934002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":845,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-12-15T17:23:55.320927+0000","last_change":"2023-12-15T17:03:32.049516+0000","last_active":"2023-12-15T17:23:55.320927+0000","last_peered":"2023-12-15T17:23:55.320927+0000","last_clean":"2023-12-15T17:23:55.320927+0000","last_became_active":"2023-12-15T17:03:32.040162+0000","last_became_peered":"2023-12-15T17:03:32.040162+0000","last_unstale":"2023-12-15T17:23:55.320927+0000","last_undegraded":"2023-12-15T17:23:55.320927+0000","last_fullsized":"2023-12-15T17:23:55.320927+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:24:16.619046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":854,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.908823+0000","last_change":"2023-12-15T17:03:29.590131+0000","last_active":"2023-12-15T17:23:56.908823+0000","last_peered":"2023-12-15T17:23:56.908823+0000","last_clean":"2023-12-15T17:23:56.908823+0000","last_became_active":"2023-12-15T17:03:29.584729+0000","last_became_peered":"2023-12-15T17:03:29.584729+0000","last_unstale":"2023-12-15T17:23:56.908823+0000","last_undegraded":"2023-12-15T17:23:56.908823+0000","last_fullsized":"2023-12-15T17:23:56.908823+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:25:55.347921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":873,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.180768+0000","last_change":"2023-12-15T17:03:27.355790+0000","last_active":"2023-12-15T17:23:58.180768+0000","last_peered":"2023-12-15T17:23:58.180768+0000","last_clean":"2023-12-15T17:23:58.180768+0000","last_became_active":"2023-12-15T17:03:27.349390+0000","last_became_peered":"2023-12-15T17:03:27.349390+0000","last_unstale":"2023-12-15T17:23:58.180768+0000","last_undegraded":"2023-12-15T17:23:58.180768+0000","last_fullsized":"2023-12-15T17:23:58.180768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:50:15.500629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":849,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.754893+0000","last_change":"2023-12-15T17:03:41.190972+0000","last_active":"2023-12-15T17:23:56.754893+0000","last_peered":"2023-12-15T17:23:56.754893+0000","last_clean":"2023-12-15T17:23:56.754893+0000","last_became_active":"2023-12-15T17:03:41.182883+0000","last_became_peered":"2023-12-15T17:03:41.182883+0000","last_unstale":"2023-12-15T17:23:56.754893+0000","last_undegraded":"2023-12-15T17:23:56.754893+0000","last_fullsized":"2023-12-15T17:23:56.754893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T00:04:20.611993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":852,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.809588+0000","last_change":"2023-12-15T17:03:27.568351+0000","last_active":"2023-12-15T17:23:56.809588+0000","last_peered":"2023-12-15T17:23:56.809588+0000","last_clean":"2023-12-15T17:23:56.809588+0000","last_became_active":"2023-12-15T17:03:27.563090+0000","last_became_peered":"2023-12-15T17:03:27.563090+0000","last_unstale":"2023-12-15T17:23:56.809588+0000","last_undegraded":"2023-12-15T17:23:56.809588+0000","last_fullsized":"2023-12-15T17:23:56.809588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:48:11.443821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":847,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.656834+0000","last_change":"2023-12-15T17:03:26.203421+0000","last_active":"2023-12-15T17:23:56.656834+0000","last_peered":"2023-12-15T17:23:56.656834+0000","last_clean":"2023-12-15T17:23:56.656834+0000","last_became_active":"2023-12-15T17:03:26.168812+0000","last_became_peered":"2023-12-15T17:03:26.168812+0000","last_unstale":"2023-12-15T17:23:56.656834+0000","last_undegraded":"2023-12-15T17:23:56.656834+0000","last_fullsized":"2023-12-15T17:23:56.656834+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:30:33.793828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":849,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.744754+0000","last_change":"2023-12-15T17:03:33.666043+0000","last_active":"2023-12-15T17:23:56.744754+0000","last_peered":"2023-12-15T17:23:56.744754+0000","last_clean":"2023-12-15T17:23:56.744754+0000","last_became_active":"2023-12-15T17:03:33.660278+0000","last_became_peered":"2023-12-15T17:03:33.660278+0000","last_unstale":"2023-12-15T17:23:56.744754+0000","last_undegraded":"2023-12-15T17:23:56.744754+0000","last_fullsized":"2023-12-15T17:23:56.744754+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:21:15.535555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":852,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.757186+0000","last_change":"2023-12-15T17:03:29.499597+0000","last_active":"2023-12-15T17:23:56.757186+0000","last_peered":"2023-12-15T17:23:56.757186+0000","last_clean":"2023-12-15T17:23:56.757186+0000","last_became_active":"2023-12-15T17:03:29.479817+0000","last_became_peered":"2023-12-15T17:03:29.479817+0000","last_unstale":"2023-12-15T17:23:56.757186+0000","last_undegraded":"2023-12-15T17:23:56.757186+0000","last_fullsized":"2023-12-15T17:23:56.757186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:29:30.677748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":870,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.247428+0000","last_change":"2023-12-15T17:03:27.798402+0000","last_active":"2023-12-15T17:23:57.247428+0000","last_peered":"2023-12-15T17:23:57.247428+0000","last_clean":"2023-12-15T17:23:57.247428+0000","last_became_active":"2023-12-15T17:03:27.764430+0000","last_became_peered":"2023-12-15T17:03:27.764430+0000","last_unstale":"2023-12-15T17:23:57.247428+0000","last_undegraded":"2023-12-15T17:23:57.247428+0000","last_fullsized":"2023-12-15T17:23:57.247428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:50:51.103963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":882,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.172032+0000","last_change":"2023-12-15T17:02:01.284924+0000","last_active":"2023-12-15T17:23:56.172032+0000","last_peered":"2023-12-15T17:23:56.172032+0000","last_clean":"2023-12-15T17:23:56.172032+0000","last_became_active":"2023-12-15T17:02:01.275938+0000","last_became_peered":"2023-12-15T17:02:01.275938+0000","last_unstale":"2023-12-15T17:23:56.172032+0000","last_undegraded":"2023-12-15T17:23:56.172032+0000","last_fullsized":"2023-12-15T17:23:56.172032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:11:11.381089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":874,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.177122+0000","last_change":"2023-12-15T17:03:29.741478+0000","last_active":"2023-12-15T17:23:58.177122+0000","last_peered":"2023-12-15T17:23:58.177122+0000","last_clean":"2023-12-15T17:23:58.177122+0000","last_became_active":"2023-12-15T17:03:29.736171+0000","last_became_peered":"2023-12-15T17:03:29.736171+0000","last_unstale":"2023-12-15T17:23:58.177122+0000","last_undegraded":"2023-12-15T17:23:58.177122+0000","last_fullsized":"2023-12-15T17:23:58.177122+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:42:10.131036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":876,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.267646+0000","last_change":"2023-12-15T17:01:59.949551+0000","last_active":"2023-12-15T17:23:57.267646+0000","last_peered":"2023-12-15T17:23:57.267646+0000","last_clean":"2023-12-15T17:23:57.267646+0000","last_became_active":"2023-12-15T17:01:59.918026+0000","last_became_peered":"2023-12-15T17:01:59.918026+0000","last_unstale":"2023-12-15T17:23:57.267646+0000","last_undegraded":"2023-12-15T17:23:57.267646+0000","last_fullsized":"2023-12-15T17:23:57.267646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:10:51.112278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":849,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.663013+0000","last_change":"2023-12-15T17:03:26.256693+0000","last_active":"2023-12-15T17:23:56.663013+0000","last_peered":"2023-12-15T17:23:56.663013+0000","last_clean":"2023-12-15T17:23:56.663013+0000","last_became_active":"2023-12-15T17:03:26.235756+0000","last_became_peered":"2023-12-15T17:03:26.235756+0000","last_unstale":"2023-12-15T17:23:56.663013+0000","last_undegraded":"2023-12-15T17:23:56.663013+0000","last_fullsized":"2023-12-15T17:23:56.663013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:08:19.767760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":882,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.147791+0000","last_change":"2023-12-15T17:02:01.380902+0000","last_active":"2023-12-15T17:23:56.147791+0000","last_peered":"2023-12-15T17:23:56.147791+0000","last_clean":"2023-12-15T17:23:56.147791+0000","last_became_active":"2023-12-15T17:02:01.375672+0000","last_became_peered":"2023-12-15T17:02:01.375672+0000","last_unstale":"2023-12-15T17:23:56.147791+0000","last_undegraded":"2023-12-15T17:23:56.147791+0000","last_fullsized":"2023-12-15T17:23:56.147791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:15:17.260715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"32'7","reported_seq":863,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.747693+0000","last_change":"2023-12-15T17:03:26.359500+0000","last_active":"2023-12-15T17:23:56.747693+0000","last_peered":"2023-12-15T17:23:56.747693+0000","last_clean":"2023-12-15T17:23:56.747693+0000","last_became_active":"2023-12-15T17:03:26.343248+0000","last_became_peered":"2023-12-15T17:03:26.343248+0000","last_unstale":"2023-12-15T17:23:56.747693+0000","last_undegraded":"2023-12-15T17:23:56.747693+0000","last_fullsized":"2023-12-15T17:23:56.747693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:31:17.788444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":881,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.185109+0000","last_change":"2023-12-15T17:01:59.938931+0000","last_active":"2023-12-15T17:23:58.185109+0000","last_peered":"2023-12-15T17:23:58.185109+0000","last_clean":"2023-12-15T17:23:58.185109+0000","last_became_active":"2023-12-15T17:01:59.908680+0000","last_became_peered":"2023-12-15T17:01:59.908680+0000","last_unstale":"2023-12-15T17:23:58.185109+0000","last_undegraded":"2023-12-15T17:23:58.185109+0000","last_fullsized":"2023-12-15T17:23:58.185109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:35:37.366491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.170131+0000","last_change":"2023-12-15T17:03:27.788302+0000","last_active":"2023-12-15T17:23:56.170131+0000","last_peered":"2023-12-15T17:23:56.170131+0000","last_clean":"2023-12-15T17:23:56.170131+0000","last_became_active":"2023-12-15T17:03:27.780434+0000","last_became_peered":"2023-12-15T17:03:27.780434+0000","last_unstale":"2023-12-15T17:23:56.170131+0000","last_undegraded":"2023-12-15T17:23:56.170131+0000","last_fullsized":"2023-12-15T17:23:56.170131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:24:22.326668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":883,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.126527+0000","last_change":"2023-12-15T17:02:01.718405+0000","last_active":"2023-12-15T17:23:56.126527+0000","last_peered":"2023-12-15T17:23:56.126527+0000","last_clean":"2023-12-15T17:23:56.126527+0000","last_became_active":"2023-12-15T17:02:01.713188+0000","last_became_peered":"2023-12-15T17:02:01.713188+0000","last_unstale":"2023-12-15T17:23:56.126527+0000","last_undegraded":"2023-12-15T17:23:56.126527+0000","last_fullsized":"2023-12-15T17:23:56.126527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:27:19.133287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":870,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.253473+0000","last_change":"2023-12-15T17:03:26.257800+0000","last_active":"2023-12-15T17:23:57.253473+0000","last_peered":"2023-12-15T17:23:57.253473+0000","last_clean":"2023-12-15T17:23:57.253473+0000","last_became_active":"2023-12-15T17:03:26.252650+0000","last_became_peered":"2023-12-15T17:03:26.252650+0000","last_unstale":"2023-12-15T17:23:57.253473+0000","last_undegraded":"2023-12-15T17:23:57.253473+0000","last_fullsized":"2023-12-15T17:23:57.253473+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:20:58.993074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":859,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.780879+0000","last_change":"2023-12-15T17:01:59.559261+0000","last_active":"2023-12-15T17:23:56.780879+0000","last_peered":"2023-12-15T17:23:56.780879+0000","last_clean":"2023-12-15T17:23:56.780879+0000","last_became_active":"2023-12-15T17:01:59.531480+0000","last_became_peered":"2023-12-15T17:01:59.531480+0000","last_unstale":"2023-12-15T17:23:56.780879+0000","last_undegraded":"2023-12-15T17:23:56.780879+0000","last_fullsized":"2023-12-15T17:23:56.780879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:51:07.527384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.146161+0000","last_change":"2023-12-15T17:03:29.375364+0000","last_active":"2023-12-15T17:23:56.146161+0000","last_peered":"2023-12-15T17:23:56.146161+0000","last_clean":"2023-12-15T17:23:56.146161+0000","last_became_active":"2023-12-15T17:03:29.361304+0000","last_became_peered":"2023-12-15T17:03:29.361304+0000","last_unstale":"2023-12-15T17:23:56.146161+0000","last_undegraded":"2023-12-15T17:23:56.146161+0000","last_fullsized":"2023-12-15T17:23:56.146161+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:51:08.058570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":882,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.102856+0000","last_change":"2023-12-15T17:02:01.240306+0000","last_active":"2023-12-15T17:23:56.102856+0000","last_peered":"2023-12-15T17:23:56.102856+0000","last_clean":"2023-12-15T17:23:56.102856+0000","last_became_active":"2023-12-15T17:02:01.208353+0000","last_became_peered":"2023-12-15T17:02:01.208353+0000","last_unstale":"2023-12-15T17:23:56.102856+0000","last_undegraded":"2023-12-15T17:23:56.102856+0000","last_fullsized":"2023-12-15T17:23:56.102856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:41:20.461821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":874,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.191435+0000","last_change":"2023-12-15T17:03:27.328921+0000","last_active":"2023-12-15T17:23:58.191435+0000","last_peered":"2023-12-15T17:23:58.191435+0000","last_clean":"2023-12-15T17:23:58.191435+0000","last_became_active":"2023-12-15T17:03:27.321338+0000","last_became_peered":"2023-12-15T17:03:27.321338+0000","last_unstale":"2023-12-15T17:23:58.191435+0000","last_undegraded":"2023-12-15T17:23:58.191435+0000","last_fullsized":"2023-12-15T17:23:58.191435+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:51:42.546700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":883,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.120609+0000","last_change":"2023-12-15T17:02:01.625426+0000","last_active":"2023-12-15T17:23:56.120609+0000","last_peered":"2023-12-15T17:23:56.120609+0000","last_clean":"2023-12-15T17:23:56.120609+0000","last_became_active":"2023-12-15T17:02:01.616517+0000","last_became_peered":"2023-12-15T17:02:01.616517+0000","last_unstale":"2023-12-15T17:23:56.120609+0000","last_undegraded":"2023-12-15T17:23:56.120609+0000","last_fullsized":"2023-12-15T17:23:56.120609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:54:17.323141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":961,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.661499+0000","last_change":"2023-12-15T17:01:52.907228+0000","last_active":"2023-12-15T17:23:56.661499+0000","last_peered":"2023-12-15T17:23:56.661499+0000","last_clean":"2023-12-15T17:23:56.661499+0000","last_became_active":"2023-12-15T17:01:52.878062+0000","last_became_peered":"2023-12-15T17:01:52.878062+0000","last_unstale":"2023-12-15T17:23:56.661499+0000","last_undegraded":"2023-12-15T17:23:56.661499+0000","last_fullsized":"2023-12-15T17:23:56.661499+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_clean_scrub_stamp":"2023-12-15T17:01:50.252516+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:47:14.717362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":850,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.753308+0000","last_change":"2023-12-15T17:03:31.083758+0000","last_active":"2023-12-15T17:23:56.753308+0000","last_peered":"2023-12-15T17:23:56.753308+0000","last_clean":"2023-12-15T17:23:56.753308+0000","last_became_active":"2023-12-15T17:03:31.075388+0000","last_became_peered":"2023-12-15T17:03:31.075388+0000","last_unstale":"2023-12-15T17:23:56.753308+0000","last_undegraded":"2023-12-15T17:23:56.753308+0000","last_fullsized":"2023-12-15T17:23:56.753308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:32:49.564294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":848,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.697108+0000","last_change":"2023-12-15T17:03:29.701214+0000","last_active":"2023-12-15T17:23:56.697108+0000","last_peered":"2023-12-15T17:23:56.697108+0000","last_clean":"2023-12-15T17:23:56.697108+0000","last_became_active":"2023-12-15T17:03:29.683464+0000","last_became_peered":"2023-12-15T17:03:29.683464+0000","last_unstale":"2023-12-15T17:23:56.697108+0000","last_undegraded":"2023-12-15T17:23:56.697108+0000","last_fullsized":"2023-12-15T17:23:56.697108+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:02:06.165049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":853,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.906574+0000","last_change":"2023-12-15T17:03:26.043197+0000","last_active":"2023-12-15T17:23:56.906574+0000","last_peered":"2023-12-15T17:23:56.906574+0000","last_clean":"2023-12-15T17:23:56.906574+0000","last_became_active":"2023-12-15T17:03:26.036270+0000","last_became_peered":"2023-12-15T17:03:26.036270+0000","last_unstale":"2023-12-15T17:23:56.906574+0000","last_undegraded":"2023-12-15T17:23:56.906574+0000","last_fullsized":"2023-12-15T17:23:56.906574+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:27:28.632284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":851,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.960741+0000","last_change":"2023-12-15T17:03:29.615007+0000","last_active":"2023-12-15T17:23:56.960741+0000","last_peered":"2023-12-15T17:23:56.960741+0000","last_clean":"2023-12-15T17:23:56.960741+0000","last_became_active":"2023-12-15T17:03:29.608805+0000","last_became_peered":"2023-12-15T17:03:29.608805+0000","last_unstale":"2023-12-15T17:23:56.960741+0000","last_undegraded":"2023-12-15T17:23:56.960741+0000","last_fullsized":"2023-12-15T17:23:56.960741+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T20:35:50.230477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":875,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.167794+0000","last_change":"2023-12-15T17:03:27.507868+0000","last_active":"2023-12-15T17:23:58.167794+0000","last_peered":"2023-12-15T17:23:58.167794+0000","last_clean":"2023-12-15T17:23:58.167794+0000","last_became_active":"2023-12-15T17:03:27.499200+0000","last_became_peered":"2023-12-15T17:03:27.499200+0000","last_unstale":"2023-12-15T17:23:58.167794+0000","last_undegraded":"2023-12-15T17:23:58.167794+0000","last_fullsized":"2023-12-15T17:23:58.167794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T20:31:29.233160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":875,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.163182+0000","last_change":"2023-12-15T17:03:28.022217+0000","last_active":"2023-12-15T17:23:58.163182+0000","last_peered":"2023-12-15T17:23:58.163182+0000","last_clean":"2023-12-15T17:23:58.163182+0000","last_became_active":"2023-12-15T17:03:28.012736+0000","last_became_peered":"2023-12-15T17:03:28.012736+0000","last_unstale":"2023-12-15T17:23:58.163182+0000","last_undegraded":"2023-12-15T17:23:58.163182+0000","last_fullsized":"2023-12-15T17:23:58.163182+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T23:37:05.516314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":870,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.245793+0000","last_change":"2023-12-15T17:03:27.802217+0000","last_active":"2023-12-15T17:23:57.245793+0000","last_peered":"2023-12-15T17:23:57.245793+0000","last_clean":"2023-12-15T17:23:57.245793+0000","last_became_active":"2023-12-15T17:03:27.791224+0000","last_became_peered":"2023-12-15T17:03:27.791224+0000","last_unstale":"2023-12-15T17:23:57.245793+0000","last_undegraded":"2023-12-15T17:23:57.245793+0000","last_fullsized":"2023-12-15T17:23:57.245793+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:54:03.758840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":851,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.326424+0000","last_change":"2023-12-15T17:03:34.018076+0000","last_active":"2023-12-15T17:23:58.326424+0000","last_peered":"2023-12-15T17:23:58.326424+0000","last_clean":"2023-12-15T17:23:58.326424+0000","last_became_active":"2023-12-15T17:03:33.897262+0000","last_became_peered":"2023-12-15T17:03:33.897262+0000","last_unstale":"2023-12-15T17:23:58.326424+0000","last_undegraded":"2023-12-15T17:23:58.326424+0000","last_fullsized":"2023-12-15T17:23:58.326424+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T23:33:26.034906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":874,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.183460+0000","last_change":"2023-12-15T17:03:28.235706+0000","last_active":"2023-12-15T17:23:58.183460+0000","last_peered":"2023-12-15T17:23:58.183460+0000","last_clean":"2023-12-15T17:23:58.183460+0000","last_became_active":"2023-12-15T17:03:28.164968+0000","last_became_peered":"2023-12-15T17:03:28.164968+0000","last_unstale":"2023-12-15T17:23:58.183460+0000","last_undegraded":"2023-12-15T17:23:58.183460+0000","last_fullsized":"2023-12-15T17:23:58.183460+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:55:52.108198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":875,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.143397+0000","last_change":"2023-12-15T17:03:29.256120+0000","last_active":"2023-12-15T17:23:56.143397+0000","last_peered":"2023-12-15T17:23:56.143397+0000","last_clean":"2023-12-15T17:23:56.143397+0000","last_became_active":"2023-12-15T17:03:29.219241+0000","last_became_peered":"2023-12-15T17:03:29.219241+0000","last_unstale":"2023-12-15T17:23:56.143397+0000","last_undegraded":"2023-12-15T17:23:56.143397+0000","last_fullsized":"2023-12-15T17:23:56.143397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:27:29.782496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":847,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.691386+0000","last_change":"2023-12-15T17:03:25.431259+0000","last_active":"2023-12-15T17:23:56.691386+0000","last_peered":"2023-12-15T17:23:56.691386+0000","last_clean":"2023-12-15T17:23:56.691386+0000","last_became_active":"2023-12-15T17:03:25.394333+0000","last_became_peered":"2023-12-15T17:03:25.394333+0000","last_unstale":"2023-12-15T17:23:56.691386+0000","last_undegraded":"2023-12-15T17:23:56.691386+0000","last_fullsized":"2023-12-15T17:23:56.691386+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:35:32.135097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":847,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.694198+0000","last_change":"2023-12-15T17:03:25.933659+0000","last_active":"2023-12-15T17:23:56.694198+0000","last_peered":"2023-12-15T17:23:56.694198+0000","last_clean":"2023-12-15T17:23:56.694198+0000","last_became_active":"2023-12-15T17:03:25.924138+0000","last_became_peered":"2023-12-15T17:03:25.924138+0000","last_unstale":"2023-12-15T17:23:56.694198+0000","last_undegraded":"2023-12-15T17:23:56.694198+0000","last_fullsized":"2023-12-15T17:23:56.694198+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:30:47.546645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509707,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58908,"kb_used_data":5512,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312932,"statfs":{"total":96636764160,"available":96576442368,"internally_reserved":0,"allocated":5644288,"data_stored":2992122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":10.289999999999999,"5min":13.026999999999999,"15min":16.065000000000001},"min":{"1min":5.0090000000000003,"5min":4.351,"15min":4.3390000000000004},"max":{"1min":20.521000000000001,"5min":151.61099999999999,"15min":199.32400000000001},"last":17.937000000000001},{"interface":"front","average":{"1min":10.064,"5min":14.347,"15min":17.850999999999999},"min":{"1min":6.2839999999999998,"5min":5.0149999999999997,"15min":4.9370000000000003},"max":{"1min":20.202999999999999,"5min":278.32799999999997,"15min":330.613},"last":21.178000000000001}]},{"osd":1,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":16.135000000000002,"5min":17.571999999999999,"15min":19.690000000000001},"min":{"1min":7.4400000000000004,"5min":6.7439999999999998,"15min":5.726},"max":{"1min":65.194999999999993,"5min":159.22399999999999,"15min":303.95400000000001},"last":7.335},{"interface":"front","average":{"1min":15.039999999999999,"5min":19.803000000000001,"15min":20.053999999999998},"min":{"1min":7.5439999999999996,"5min":6.923,"15min":6.4610000000000003},"max":{"1min":55.744,"5min":326.125,"15min":326.125},"last":21.805}]},{"osd":3,"last update":"Fri Dec 15 17:23:02 2023","interfaces":[{"interface":"back","average":{"1min":10.766999999999999,"5min":17.425999999999998,"15min":23.780000000000001},"min":{"1min":6.6740000000000004,"5min":5.944,"15min":5.9050000000000002},"max":{"1min":30.068000000000001,"5min":309.05399999999997,"15min":313.50400000000002},"last":7.7539999999999996},{"interface":"front","average":{"1min":15.036,"5min":20.998999999999999,"15min":25.573},"min":{"1min":7.476,"5min":7.2480000000000002,"15min":6.1449999999999996},"max":{"1min":28.457999999999998,"5min":372.85199999999998,"15min":372.85199999999998},"last":20.539999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":15.255000000000001,"5min":23.908000000000001,"15min":24.216000000000001},"min":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":6.5359999999999996},"max":{"1min":26.652000000000001,"5min":397.04000000000002,"15min":397.04000000000002},"last":28.001000000000001},{"interface":"front","average":{"1min":16.533999999999999,"5min":24.268999999999998,"15min":25.311},"min":{"1min":8.5980000000000008,"5min":8.5980000000000008,"15min":7.7809999999999997},"max":{"1min":31.385999999999999,"5min":217.01599999999999,"15min":217.01599999999999},"last":16.015000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:08 2023","interfaces":[{"interface":"back","average":{"1min":17.076000000000001,"5min":22.579000000000001,"15min":34.334000000000003},"min":{"1min":8.2629999999999999,"5min":8.2629999999999999,"15min":7.7839999999999998},"max":{"1min":29.724,"5min":196.75700000000001,"15min":493.322},"last":17.254000000000001},{"interface":"front","average":{"1min":13.827,"5min":18.082000000000001,"15min":30.268000000000001},"min":{"1min":7.5880000000000001,"5min":5.9160000000000004,"15min":5.9160000000000004},"max":{"1min":25.835000000000001,"5min":280.45400000000001,"15min":497.67200000000003},"last":22.481999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:49 2023","interfaces":[{"interface":"back","average":{"1min":16.161000000000001,"5min":22.856000000000002,"15min":25.489999999999998},"min":{"1min":10.462,"5min":9.4039999999999999,"15min":8.5969999999999995},"max":{"1min":25.347999999999999,"5min":289.68400000000003,"15min":289.68400000000003},"last":36.061},{"interface":"front","average":{"1min":14.811999999999999,"5min":21.321999999999999,"15min":22.838999999999999},"min":{"1min":9.6989999999999998,"5min":7.8200000000000003,"15min":6.9409999999999998},"max":{"1min":22.036000000000001,"5min":274.80099999999999,"15min":274.80099999999999},"last":26.370000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:02 2023","interfaces":[{"interface":"back","average":{"1min":19.841999999999999,"5min":24.600999999999999,"15min":31.303000000000001},"min":{"1min":9.3930000000000007,"5min":9.3930000000000007,"15min":8.5820000000000007},"max":{"1min":32.054000000000002,"5min":278.98000000000002,"15min":613.21299999999997},"last":29.344999999999999},{"interface":"front","average":{"1min":17.178999999999998,"5min":22.213999999999999,"15min":29.68},"min":{"1min":7.8360000000000003,"5min":7.8360000000000003,"15min":7.8360000000000003},"max":{"1min":29.925999999999998,"5min":275.72300000000001,"15min":613.91099999999994},"last":23.212}]}]},{"osd":1,"up_from":15,"seq":64424509710,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81508,"kb_used_data":5904,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290332,"statfs":{"total":96636764160,"available":96553299968,"internally_reserved":0,"allocated":6045696,"data_stored":3389962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":12.414999999999999,"5min":16.533000000000001,"15min":22.98},"min":{"1min":5.2759999999999998,"5min":4.4870000000000001,"15min":4.383},"max":{"1min":36.951999999999998,"5min":94.881,"15min":452.79500000000002},"last":7.0010000000000003},{"interface":"front","average":{"1min":16.593,"5min":15.000999999999999,"15min":17.280000000000001},"min":{"1min":5.2960000000000003,"5min":5.0419999999999998,"15min":4.3449999999999998},"max":{"1min":90.009,"5min":127.498,"15min":196.59200000000001},"last":10.116}]},{"osd":2,"last update":"Fri Dec 15 17:23:03 2023","interfaces":[{"interface":"back","average":{"1min":24.509,"5min":19.314,"15min":25.472000000000001},"min":{"1min":6.4480000000000004,"5min":5.7380000000000004,"15min":5.5190000000000001},"max":{"1min":276.47000000000003,"5min":276.47000000000003,"15min":482.72800000000001},"last":18.846},{"interface":"front","average":{"1min":25.614000000000001,"5min":22.815000000000001,"15min":31.640999999999998},"min":{"1min":8.2119999999999997,"5min":6.5629999999999997,"15min":6.1699999999999999},"max":{"1min":261.74400000000003,"5min":261.74400000000003,"15min":454.017},"last":17.173999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":19.303000000000001,"5min":20.509,"15min":28.212},"min":{"1min":7.1100000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"max":{"1min":110.723,"5min":147.37100000000001,"15min":488.23700000000002},"last":33.832000000000001},{"interface":"front","average":{"1min":16.472000000000001,"5min":21.302,"15min":26.646000000000001},"min":{"1min":10.347,"5min":9.1549999999999994,"15min":6.4119999999999999},"max":{"1min":30.919,"5min":110.056,"15min":333.53199999999998},"last":28.303000000000001}]},{"osd":4,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":13.672000000000001,"5min":18.148,"15min":26.068999999999999},"min":{"1min":7.1959999999999997,"5min":6.2629999999999999,"15min":5.8319999999999999},"max":{"1min":53.085999999999999,"5min":143.24000000000001,"15min":438.75099999999998},"last":7.8970000000000002},{"interface":"front","average":{"1min":18.068999999999999,"5min":21.373000000000001,"15min":31.120000000000001},"min":{"1min":10.132999999999999,"5min":8.4510000000000005,"15min":8.4339999999999993},"max":{"1min":41.640000000000001,"5min":78.664000000000001,"15min":443.58300000000003},"last":27.532}]},{"osd":5,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":22.023,"5min":20.832000000000001,"15min":36.176000000000002},"min":{"1min":8.4710000000000001,"5min":6.8369999999999997,"15min":6.8369999999999997},"max":{"1min":112.105,"5min":113.04300000000001,"15min":451.70999999999998},"last":8.9260000000000002},{"interface":"front","average":{"1min":31.739999999999998,"5min":27.744,"15min":29.515999999999998},"min":{"1min":12.116,"5min":9.8840000000000003,"15min":6.6500000000000004},"max":{"1min":209.96600000000001,"5min":209.96600000000001,"15min":264.46899999999999},"last":31.350999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":21.939,"5min":22.827999999999999,"15min":31.187999999999999},"min":{"1min":12.606,"5min":8.6809999999999992,"15min":7.5629999999999997},"max":{"1min":90.926000000000002,"5min":162.06200000000001,"15min":455.43000000000001},"last":34.859000000000002},{"interface":"front","average":{"1min":19.638999999999999,"5min":21.096,"15min":28.334},"min":{"1min":9.1400000000000006,"5min":7.3399999999999999,"15min":6.7249999999999996},"max":{"1min":94.715999999999994,"5min":169.34,"15min":314.589},"last":18.175000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":21.369,"5min":26.395,"15min":36.750999999999998},"min":{"1min":10.898999999999999,"5min":10.691000000000001,"15min":10.561999999999999},"max":{"1min":59.499000000000002,"5min":132.423,"15min":453.04300000000001},"last":28.978000000000002},{"interface":"front","average":{"1min":20.079999999999998,"5min":21.163,"15min":28.721},"min":{"1min":10.175000000000001,"5min":10.175000000000001,"15min":8.6110000000000007},"max":{"1min":63.219999999999999,"5min":77.674000000000007,"15min":239.51499999999999},"last":35.472999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58892,"kb_used_data":5496,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312948,"statfs":{"total":96636764160,"available":96576458752,"internally_reserved":0,"allocated":5627904,"data_stored":2985572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":13.765000000000001,"5min":13.314,"15min":18.777000000000001},"min":{"1min":6.2359999999999998,"5min":4.3639999999999999,"15min":4.1429999999999998},"max":{"1min":80.552999999999997,"5min":80.552999999999997,"15min":375.77800000000002},"last":10.553000000000001},{"interface":"front","average":{"1min":17.832999999999998,"5min":17.524999999999999,"15min":23.969000000000001},"min":{"1min":6.0199999999999996,"5min":4.8170000000000002,"15min":4.8170000000000002},"max":{"1min":94.995999999999995,"5min":94.995999999999995,"15min":556.42700000000002},"last":7.6689999999999996}]},{"osd":2,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":20.041,"5min":19.704999999999998,"15min":21.385999999999999},"min":{"1min":9.3889999999999993,"5min":6.2939999999999996,"15min":6.2939999999999996},"max":{"1min":99.216999999999999,"5min":248.864,"15min":396.13400000000001},"last":9.2669999999999995},{"interface":"front","average":{"1min":16.388999999999999,"5min":21.504999999999999,"15min":25.138999999999999},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":5.9790000000000001},"max":{"1min":75.944000000000003,"5min":234.161,"15min":354.649},"last":11.680999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":19.158000000000001,"5min":21.491,"15min":22.245000000000001},"min":{"1min":7.6159999999999997,"5min":6.1740000000000004,"15min":6.0830000000000002},"max":{"1min":96.188000000000002,"5min":99.453000000000003,"15min":413.00599999999997},"last":93.516000000000005},{"interface":"front","average":{"1min":15.550000000000001,"5min":19.518999999999998,"15min":24.094999999999999},"min":{"1min":6.0640000000000001,"5min":6.04,"15min":6.04},"max":{"1min":101.57299999999999,"5min":106.357,"15min":526.98599999999999},"last":84.674000000000007}]},{"osd":4,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":21.193999999999999,"5min":24.096,"15min":24.780999999999999},"min":{"1min":7.1829999999999998,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":119.754,"5min":266.279,"15min":389.75700000000001},"last":7.1539999999999999},{"interface":"front","average":{"1min":17.59,"5min":22.911999999999999,"15min":25.393000000000001},"min":{"1min":8.4700000000000006,"5min":8.4700000000000006,"15min":7.3520000000000003},"max":{"1min":37.862000000000002,"5min":227.19200000000001,"15min":436.572},"last":12.255000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":17.655000000000001,"5min":22.420000000000002,"15min":26.187999999999999},"min":{"1min":7.8220000000000001,"5min":7.5359999999999996,"15min":7.3540000000000001},"max":{"1min":67.956999999999994,"5min":121.137,"15min":449.81299999999999},"last":224.11099999999999},{"interface":"front","average":{"1min":20.068000000000001,"5min":23.791,"15min":27.331},"min":{"1min":10.81,"5min":7.7439999999999998,"15min":6.2539999999999996},"max":{"1min":65.381,"5min":109.90000000000001,"15min":429.75299999999999},"last":227.60900000000001}]},{"osd":6,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":19.919,"5min":23.132999999999999,"15min":25.440999999999999},"min":{"1min":10.682,"5min":8.6760000000000002,"15min":7.4699999999999998},"max":{"1min":91.619,"5min":93.299000000000007,"15min":469.86799999999999},"last":15.897},{"interface":"front","average":{"1min":23.565000000000001,"5min":24.164000000000001,"15min":27.797999999999998},"min":{"1min":12.039,"5min":9.2899999999999991,"15min":8.4740000000000002},"max":{"1min":102.664,"5min":102.664,"15min":373.93599999999998},"last":13.752000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":17.504999999999999,"5min":21.597999999999999,"15min":29.437999999999999},"min":{"1min":8.3109999999999999,"5min":7.0449999999999999,"15min":7.0449999999999999},"max":{"1min":77.521000000000001,"5min":151.999,"15min":416.24400000000003},"last":9.9390000000000001},{"interface":"front","average":{"1min":24.347000000000001,"5min":26.122,"15min":27.091000000000001},"min":{"1min":11.058,"5min":9.8610000000000007,"15min":7.7160000000000002},"max":{"1min":100.611,"5min":163.09700000000001,"15min":410.04300000000001},"last":14.868}]}]},{"osd":3,"up_from":15,"seq":64424509706,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76944,"kb_used_data":5508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294896,"statfs":{"total":96636764160,"available":96557973504,"internally_reserved":0,"allocated":5640192,"data_stored":2992117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":8.1609999999999996,"5min":13.029,"15min":18.565999999999999},"min":{"1min":5.3899999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"max":{"1min":14.663,"5min":132.791,"15min":252.483},"last":4.7389999999999999},{"interface":"front","average":{"1min":8.7210000000000001,"5min":13.803000000000001,"15min":18.956},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":24.484999999999999,"5min":138.03800000000001,"15min":235.10300000000001},"last":7.641}]},{"osd":1,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":10.631,"5min":13.651,"15min":22},"min":{"1min":7.4589999999999996,"5min":4.9390000000000001,"15min":4.9390000000000001},"max":{"1min":19.66,"5min":79.147000000000006,"15min":624.94100000000003},"last":21.771999999999998},{"interface":"front","average":{"1min":14.163,"5min":17.541,"15min":24.814},"min":{"1min":8.8680000000000003,"5min":7.4770000000000003,"15min":6.9740000000000002},"max":{"1min":22.484000000000002,"5min":86.152000000000001,"15min":635.13199999999995},"last":32.859999999999999}]},{"osd":2,"last update":"Fri Dec 15 17:23:45 2023","interfaces":[{"interface":"back","average":{"1min":14.015000000000001,"5min":17.581,"15min":22.074999999999999},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":60.210000000000001,"5min":122.60899999999999,"15min":286.22699999999998},"last":10.489000000000001},{"interface":"front","average":{"1min":13.85,"5min":17.390000000000001,"15min":25.114000000000001},"min":{"1min":6.931,"5min":6.931,"15min":6.5380000000000003},"max":{"1min":73.950000000000003,"5min":238.096,"15min":573.05700000000002},"last":14.992000000000001}]},{"osd":4,"last update":"Fri Dec 15 17:23:00 2023","interfaces":[{"interface":"back","average":{"1min":16.614000000000001,"5min":19.768999999999998,"15min":28.959},"min":{"1min":7.5019999999999998,"5min":7.5019999999999998,"15min":7.1159999999999997},"max":{"1min":60.703000000000003,"5min":228.708,"15min":628.97299999999996},"last":9.4979999999999993},{"interface":"front","average":{"1min":19.669,"5min":24.192,"15min":34.237000000000002},"min":{"1min":13.58,"5min":11.423,"15min":8.8239999999999998},"max":{"1min":29.742999999999999,"5min":230.202,"15min":637.40200000000004},"last":18.204999999999998}]},{"osd":5,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":14.532,"5min":20.297999999999998,"15min":26.827999999999999},"min":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.782},"max":{"1min":32.411000000000001,"5min":144.59200000000001,"15min":752.79499999999996},"last":8.6159999999999997},{"interface":"front","average":{"1min":16.890999999999998,"5min":21.039000000000001,"15min":25.788},"min":{"1min":9.5,"5min":9.2129999999999992,"15min":6.0119999999999996},"max":{"1min":45.607999999999997,"5min":145.405,"15min":263.79500000000002},"last":15.510999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":16.896000000000001,"5min":21.753,"15min":27.977},"min":{"1min":9.3599999999999994,"5min":8.0150000000000006,"15min":7.5819999999999999},"max":{"1min":40.414999999999999,"5min":134.255,"15min":498.464},"last":11.379},{"interface":"front","average":{"1min":21.16,"5min":24.161000000000001,"15min":31.126999999999999},"min":{"1min":12.519,"5min":10.362,"15min":8.6560000000000006},"max":{"1min":60.975000000000001,"5min":153.113,"15min":636.16399999999999},"last":20.210000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":17.704999999999998,"5min":23.234000000000002,"15min":31.463000000000001},"min":{"1min":11.615,"5min":10.316000000000001,"15min":8.4930000000000003},"max":{"1min":25.109000000000002,"5min":202.89500000000001,"15min":572.08699999999999},"last":16.465},{"interface":"front","average":{"1min":19.895,"5min":24.616,"15min":31.809999999999999},"min":{"1min":13.664999999999999,"5min":13.173999999999999,"15min":10.406000000000001},"max":{"1min":32.581000000000003,"5min":156.91900000000001,"15min":624.04100000000005},"last":17.247}]}]},{"osd":7,"up_from":14,"seq":60129542411,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76948,"kb_used_data":5508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294892,"statfs":{"total":96636764160,"available":96557969408,"internally_reserved":0,"allocated":5640192,"data_stored":2992117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":63,"apply_latency_ms":63,"commit_latency_ns":63000000,"apply_latency_ns":63000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":11.852,"5min":12.436999999999999,"15min":16.131},"min":{"1min":5.0750000000000002,"5min":4.3810000000000002,"15min":4.3259999999999996},"max":{"1min":37.542000000000002,"5min":91.055999999999997,"15min":161.38499999999999},"last":21.841000000000001},{"interface":"front","average":{"1min":12.800000000000001,"5min":12.664999999999999,"15min":16.463999999999999},"min":{"1min":6.5910000000000002,"5min":4.8339999999999996,"15min":4.8339999999999996},"max":{"1min":64.691999999999993,"5min":93.209999999999994,"15min":142.673},"last":7.3879999999999999}]},{"osd":1,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":20.167000000000002,"5min":18.283000000000001,"15min":22.882000000000001},"min":{"1min":7.2480000000000002,"5min":6.3550000000000004,"15min":5.7629999999999999},"max":{"1min":74.025999999999996,"5min":143.17599999999999,"15min":275.99299999999999},"last":15.551},{"interface":"front","average":{"1min":15.464,"5min":14.946,"15min":21.763999999999999},"min":{"1min":5.9749999999999996,"5min":5.7960000000000003,"15min":5.2649999999999997},"max":{"1min":81.099999999999994,"5min":88.022999999999996,"15min":342.22000000000003},"last":16.494}]},{"osd":2,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":20.966999999999999,"5min":17.347000000000001,"15min":19.687000000000001},"min":{"1min":8.2509999999999994,"5min":7.383,"15min":6.0670000000000002},"max":{"1min":86.772000000000006,"5min":95.926000000000002,"15min":131.81299999999999},"last":26.233000000000001},{"interface":"front","average":{"1min":16.437000000000001,"5min":14.654999999999999,"15min":19.41},"min":{"1min":6.9470000000000001,"5min":6.266,"15min":6.266},"max":{"1min":84.876000000000005,"5min":84.876000000000005,"15min":117.864},"last":23.754999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":20.248000000000001,"5min":20.637,"15min":24.939},"min":{"1min":6.5330000000000004,"5min":6.5330000000000004,"15min":6.4370000000000003},"max":{"1min":118.39400000000001,"5min":118.39400000000001,"15min":390.91000000000003},"last":19.044},{"interface":"front","average":{"1min":20.558,"5min":21.027999999999999,"15min":26.327999999999999},"min":{"1min":8.1479999999999997,"5min":7.4260000000000002,"15min":5.7320000000000002},"max":{"1min":138.559,"5min":138.559,"15min":244.13},"last":24.443999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:23:51 2023","interfaces":[{"interface":"back","average":{"1min":24.532,"5min":22.248999999999999,"15min":24.129999999999999},"min":{"1min":7.3529999999999998,"5min":7.3529999999999998,"15min":7.3529999999999998},"max":{"1min":144.52099999999999,"5min":144.52099999999999,"15min":245.65600000000001},"last":17.129000000000001},{"interface":"front","average":{"1min":25.827999999999999,"5min":24.245999999999999,"15min":23.055},"min":{"1min":9.4199999999999999,"5min":7.726,"15min":6.9139999999999997},"max":{"1min":143.92099999999999,"5min":189.988,"15min":193.07400000000001},"last":21.242000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":17.966000000000001,"5min":19.504000000000001,"15min":25.530000000000001},"min":{"1min":7.7699999999999996,"5min":7.7699999999999996,"15min":7.7699999999999996},"max":{"1min":69.656999999999996,"5min":109.459,"15min":198.96000000000001},"last":22.948},{"interface":"front","average":{"1min":21.565999999999999,"5min":20.701000000000001,"15min":27.978999999999999},"min":{"1min":8.7959999999999994,"5min":7.6189999999999998,"15min":7.6189999999999998},"max":{"1min":145.94,"5min":165.97999999999999,"15min":266.33699999999999},"last":17.718}]},{"osd":6,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":24.082000000000001,"5min":22.300000000000001,"15min":30.507000000000001},"min":{"1min":11.667999999999999,"5min":7.952,"15min":7.8949999999999996},"max":{"1min":112.369,"5min":113.128,"15min":380.62700000000001},"last":25.324999999999999},{"interface":"front","average":{"1min":23.890999999999998,"5min":21.359000000000002,"15min":26.186},"min":{"1min":8.3550000000000004,"5min":8.3550000000000004,"15min":7.2270000000000003},"max":{"1min":108.03,"5min":108.03,"15min":226.91499999999999},"last":18.34}]}]},{"osd":6,"up_from":14,"seq":60129542412,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76968,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294872,"statfs":{"total":96636764160,"available":96557948928,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":10.004,"5min":12.342000000000001,"15min":18.960000000000001},"min":{"1min":4.4269999999999996,"5min":4.133,"15min":4.133},"max":{"1min":25.085999999999999,"5min":128.82499999999999,"15min":223.15199999999999},"last":7.9279999999999999},{"interface":"front","average":{"1min":9.6329999999999991,"5min":14.333,"15min":18.896000000000001},"min":{"1min":6.2009999999999996,"5min":5.3490000000000002,"15min":4.6920000000000002},"max":{"1min":17.561,"5min":120.85299999999999,"15min":190.249},"last":11.704000000000001}]},{"osd":1,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":12.476000000000001,"5min":18.774000000000001,"15min":22.052},"min":{"1min":7.2279999999999998,"5min":5.1319999999999997,"15min":5.0449999999999999},"max":{"1min":24.771999999999998,"5min":177.15100000000001,"15min":292.72199999999998},"last":9.8870000000000005},{"interface":"front","average":{"1min":14.473000000000001,"5min":19.847999999999999,"15min":25.225000000000001},"min":{"1min":7.6120000000000001,"5min":6.4390000000000001,"15min":5.0309999999999997},"max":{"1min":41.270000000000003,"5min":174.02199999999999,"15min":422.36099999999999},"last":14.478}]},{"osd":2,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":17.154,"5min":17.030000000000001,"15min":19.913},"min":{"1min":7.7169999999999996,"5min":6.2850000000000001,"15min":5.2699999999999996},"max":{"1min":66.974000000000004,"5min":131.71000000000001,"15min":131.863},"last":17.501999999999999},{"interface":"front","average":{"1min":13.039,"5min":19.218,"15min":23.687000000000001},"min":{"1min":6.9530000000000003,"5min":6.9530000000000003,"15min":6.7439999999999998},"max":{"1min":65.632000000000005,"5min":112.548,"15min":189.417},"last":10.91}]},{"osd":3,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":21.212,"5min":24.562000000000001,"15min":27.657},"min":{"1min":7.7279999999999998,"5min":7.7279999999999998,"15min":7.7279999999999998},"max":{"1min":77.271000000000001,"5min":263.45299999999997,"15min":263.45299999999997},"last":16.798999999999999},{"interface":"front","average":{"1min":24.375,"5min":27.704000000000001,"15min":30.245999999999999},"min":{"1min":11.551,"5min":10.195,"15min":6.9100000000000001},"max":{"1min":83.891999999999996,"5min":264.54000000000002,"15min":356.815},"last":26.609000000000002}]},{"osd":4,"last update":"Fri Dec 15 17:23:25 2023","interfaces":[{"interface":"back","average":{"1min":17.318000000000001,"5min":20.620000000000001,"15min":24.326000000000001},"min":{"1min":6.4020000000000001,"5min":6.4020000000000001,"15min":6.4020000000000001},"max":{"1min":118.277,"5min":122.881,"15min":224.96299999999999},"last":31.928000000000001},{"interface":"front","average":{"1min":16.988,"5min":21.347000000000001,"15min":26.774000000000001},"min":{"1min":6.9100000000000001,"5min":6.9100000000000001,"15min":6.9100000000000001},"max":{"1min":99.513000000000005,"5min":127.607,"15min":266.88400000000001},"last":28.789999999999999}]},{"osd":5,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":15.765000000000001,"5min":21.884,"15min":28.344000000000001},"min":{"1min":7.9089999999999998,"5min":7.601,"15min":7.0369999999999999},"max":{"1min":40.454999999999998,"5min":164.62899999999999,"15min":349.983},"last":29.637},{"interface":"front","average":{"1min":14.981999999999999,"5min":21.09,"15min":25.739000000000001},"min":{"1min":9.1750000000000007,"5min":7.4530000000000003,"15min":6.8449999999999998},"max":{"1min":26.895,"5min":115.54900000000001,"15min":324.95299999999997},"last":12.641}]},{"osd":7,"last update":"Fri Dec 15 17:23:47 2023","interfaces":[{"interface":"back","average":{"1min":15.082000000000001,"5min":22.818999999999999,"15min":28.507000000000001},"min":{"1min":9.4640000000000004,"5min":8.2110000000000003,"15min":6.8520000000000003},"max":{"1min":38.795999999999999,"5min":137.59700000000001,"15min":268.66500000000002},"last":15.759},{"interface":"front","average":{"1min":18.492999999999999,"5min":26.584,"15min":33.052999999999997},"min":{"1min":12.459,"5min":8.8849999999999998,"15min":8.8849999999999998},"max":{"1min":48.68,"5min":135.74199999999999,"15min":318.56900000000002},"last":18.266999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542413,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81528,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290312,"statfs":{"total":96636764160,"available":96553279488,"internally_reserved":0,"allocated":6066176,"data_stored":3403689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":10.813000000000001,"5min":18.579999999999998,"15min":19.327999999999999},"min":{"1min":5.0339999999999998,"5min":4.7380000000000004,"15min":4.2859999999999996},"max":{"1min":24.373000000000001,"5min":209.15799999999999,"15min":223.69499999999999},"last":5.2530000000000001},{"interface":"front","average":{"1min":10.352,"5min":17.178999999999998,"15min":19.341999999999999},"min":{"1min":4.2590000000000003,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":17.353000000000002,"5min":183.93899999999999,"15min":226.76300000000001},"last":8.766}]},{"osd":1,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":13.563000000000001,"5min":17.940000000000001,"15min":22.196000000000002},"min":{"1min":6.6539999999999999,"5min":5.1260000000000003,"15min":4.7839999999999998},"max":{"1min":63.030999999999999,"5min":137.07499999999999,"15min":340.32999999999998},"last":7.8949999999999996},{"interface":"front","average":{"1min":18.015999999999998,"5min":20.047999999999998,"15min":24.577000000000002},"min":{"1min":8.7149999999999999,"5min":5.7850000000000001,"15min":5.6550000000000002},"max":{"1min":57.795000000000002,"5min":169.648,"15min":326.375},"last":11.550000000000001}]},{"osd":2,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":17.568000000000001,"5min":20.483000000000001,"15min":22.577000000000002},"min":{"1min":6.6630000000000003,"5min":6.6630000000000003,"15min":5.3979999999999997},"max":{"1min":92.230000000000004,"5min":174.09100000000001,"15min":240.00999999999999},"last":15.416},{"interface":"front","average":{"1min":21.774999999999999,"5min":24.068000000000001,"15min":25.559000000000001},"min":{"1min":6.7690000000000001,"5min":5.569,"15min":5.569},"max":{"1min":128.98099999999999,"5min":228.31899999999999,"15min":371.30099999999999},"last":10.891}]},{"osd":3,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":25.259,"5min":25.724,"15min":28.169},"min":{"1min":9.2319999999999993,"5min":6.5999999999999996,"15min":6.5999999999999996},"max":{"1min":100.711,"5min":177.51300000000001,"15min":321.26100000000002},"last":10.199},{"interface":"front","average":{"1min":25.388999999999999,"5min":26.262,"15min":27.506},"min":{"1min":7.3799999999999999,"5min":7.101,"15min":6.8730000000000002},"max":{"1min":101.599,"5min":172.887,"15min":284.12599999999998},"last":12.177}]},{"osd":4,"last update":"Fri Dec 15 17:23:40 2023","interfaces":[{"interface":"back","average":{"1min":18.481000000000002,"5min":29.667999999999999,"15min":27.547000000000001},"min":{"1min":8.3559999999999999,"5min":8.3559999999999999,"15min":6.8600000000000003},"max":{"1min":61.365000000000002,"5min":229.077,"15min":376.39800000000002},"last":13.378},{"interface":"front","average":{"1min":19.219000000000001,"5min":28.552,"15min":27.09},"min":{"1min":10.989000000000001,"5min":7.8840000000000003,"15min":6.2789999999999999},"max":{"1min":55.509999999999998,"5min":211.03100000000001,"15min":338.56400000000002},"last":16.888999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":25.266999999999999,"15min":27.039000000000001},"min":{"1min":12.606999999999999,"5min":7.8079999999999998,"15min":7.0789999999999997},"max":{"1min":57.170000000000002,"5min":215.21299999999999,"15min":378.327},"last":16.372},{"interface":"front","average":{"1min":21.053000000000001,"5min":29.216999999999999,"15min":28.443000000000001},"min":{"1min":9.3049999999999997,"5min":8.8900000000000006,"15min":6.6369999999999996},"max":{"1min":62.055999999999997,"5min":228.58799999999999,"15min":407.16500000000002},"last":14.061}]},{"osd":7,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":19.960999999999999,"5min":24.295000000000002,"15min":28.638999999999999},"min":{"1min":12.951000000000001,"5min":10.864000000000001,"15min":7.7599999999999998},"max":{"1min":39.515999999999998,"5min":151.386,"15min":292.82299999999998},"last":17.649000000000001},{"interface":"front","average":{"1min":20.140000000000001,"5min":27.221,"15min":30.507000000000001},"min":{"1min":11.416,"5min":9.2170000000000005,"15min":7.8650000000000002},"max":{"1min":28.832999999999998,"5min":190.74600000000001,"15min":312.73599999999999},"last":18.655000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542413,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58904,"kb_used_data":5508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312936,"statfs":{"total":96636764160,"available":96576446464,"internally_reserved":0,"allocated":5640192,"data_stored":2992117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":20.422000000000001,"5min":13.071,"15min":15.058},"min":{"1min":4.7779999999999996,"5min":4.4560000000000004,"15min":4.1289999999999996},"max":{"1min":62.142000000000003,"5min":62.142000000000003,"15min":193.11500000000001},"last":10.169},{"interface":"front","average":{"1min":27.135999999999999,"5min":13.522,"15min":16.253},"min":{"1min":5.9699999999999998,"5min":4.4109999999999996,"15min":4.3719999999999999},"max":{"1min":96.224000000000004,"5min":96.224000000000004,"15min":235.09800000000001},"last":6.1749999999999998}]},{"osd":1,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":26.41,"5min":16.433,"15min":19.759},"min":{"1min":8.8640000000000008,"5min":5.2199999999999998,"15min":4.9930000000000003},"max":{"1min":86.936999999999998,"5min":86.936999999999998,"15min":289.11099999999999},"last":14.173999999999999},{"interface":"front","average":{"1min":22.891999999999999,"5min":18.584,"15min":21.152999999999999},"min":{"1min":7.3520000000000003,"5min":6.2549999999999999,"15min":6.2249999999999996},"max":{"1min":83.016999999999996,"5min":130.535,"15min":218.32400000000001},"last":9.3599999999999994}]},{"osd":2,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":35.674999999999997,"5min":20.347999999999999,"15min":18.050000000000001},"min":{"1min":8.2569999999999997,"5min":6.8179999999999996,"15min":5.5880000000000001},"max":{"1min":163.31399999999999,"5min":163.31399999999999,"15min":163.31399999999999},"last":25.213999999999999},{"interface":"front","average":{"1min":31.010000000000002,"5min":19.277999999999999,"15min":18.506},"min":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":5.4690000000000003},"max":{"1min":134.78700000000001,"5min":134.78700000000001,"15min":140.416},"last":11.577}]},{"osd":3,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":26.495000000000001,"5min":23.353000000000002,"15min":21.949000000000002},"min":{"1min":11.071999999999999,"5min":8.9169999999999998,"15min":7.2450000000000001},"max":{"1min":89.188000000000002,"5min":230.505,"15min":230.505},"last":24.515000000000001},{"interface":"front","average":{"1min":22.827000000000002,"5min":19.329999999999998,"15min":21.117999999999999},"min":{"1min":9.0299999999999994,"5min":6.4210000000000003,"15min":6.4210000000000003},"max":{"1min":84.497,"5min":136.66499999999999,"15min":203.99100000000001},"last":12.701000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":19.713000000000001,"5min":19.503,"15min":21.353999999999999},"min":{"1min":7.7530000000000001,"5min":6.9930000000000003,"15min":6.9930000000000003},"max":{"1min":87.989999999999995,"5min":123.71599999999999,"15min":162.02099999999999},"last":13.314},{"interface":"front","average":{"1min":23.210999999999999,"5min":23.539999999999999,"15min":24.937999999999999},"min":{"1min":10.042999999999999,"5min":10.042999999999999,"15min":7.1130000000000004},"max":{"1min":89.099999999999994,"5min":159.16999999999999,"15min":162.922},"last":28.855}]},{"osd":6,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":25.030000000000001,"5min":19.335999999999999,"15min":22.289999999999999},"min":{"1min":7.5579999999999998,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":84.040000000000006,"5min":85.606999999999999,"15min":744.33199999999999},"last":30.443999999999999},{"interface":"front","average":{"1min":38.479999999999997,"5min":26.033000000000001,"15min":25.606000000000002},"min":{"1min":14.532999999999999,"5min":10.544,"15min":9.1799999999999997},"max":{"1min":157.441,"5min":157.441,"15min":251.19200000000001},"last":29.472000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":31.425999999999998,"5min":25.463000000000001,"15min":26.760999999999999},"min":{"1min":17.059000000000001,"5min":11.148999999999999,"15min":9.7330000000000005},"max":{"1min":91.006,"5min":144.72,"15min":202.46600000000001},"last":27.286999999999999},{"interface":"front","average":{"1min":21.945,"5min":18.006,"15min":20.908000000000001},"min":{"1min":9.6690000000000005,"5min":7.5860000000000003,"15min":6.5430000000000001},"max":{"1min":78.950000000000003,"5min":85.629999999999995,"15min":199.01400000000001},"last":10.967000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-15T17:23:59.959 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-15T17:23:59.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T17:24:00.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.065+0000 7fe088141640 10 monclient: get_monmap_and_config 2023-12-15T17:24:00.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.065+0000 7fe088141640 10 monclient: build_initial_monmap 2023-12-15T17:24:00.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.065+0000 7fe088141640 10 monclient: monmap: 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:00.066589+0000 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:00.066589+0000 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.065+0000 7fe088141640 1 Processor -- start 2023-12-15T17:24:00.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.065+0000 7fe088141640 1 -- start start 2023-12-15T17:24:00.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.065+0000 7fe088141640 10 monclient: init 2023-12-15T17:24:00.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:00.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-12-15T17:24:00.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 10 monclient(hunting): picked mon.noname-b con 0x7fe0800a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:00.069 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe0800a8e10 0x7fe080142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 10 monclient(hunting): picked mon.noname-c con 0x7fe0800a8e10 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe0801454a0 0x7fe080145930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 10 monclient(hunting): picked mon.noname-a con 0x7fe0801454a0 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0800aaa10 con 0x7fe0801454a0 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0800aac10 con 0x7fe0800a8e10 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800aaef0 con 0x7fe0800a8a30 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.066+0000 7fe088141640 10 monclient(hunting): authenticate will time out at 2744.383285s 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.067+0000 7fe085edd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.067+0000 7fe0856dc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe0800a8e10 0x7fe080142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:00.070 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.067+0000 7fe085edd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42254/0 (socket says 172.21.15.69:42254) 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.067+0000 7fe085edd640 1 -- 172.21.15.69:0/1703829824 learned_addr learned my addr 172.21.15.69:0/1703829824 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.067+0000 7fe085edd640 10 monclient(hunting): get_auth_request con 0x7fe0800a8a30 auth_method 0 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.067+0000 7fe085edd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.067+0000 7fe085edd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.067+0000 7fe085edd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.068+0000 7fe0856dc640 10 monclient(hunting): get_auth_request con 0x7fe0800a8e10 auth_method 0 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.068+0000 7fe0856dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.068+0000 7fe0856dc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.068+0000 7fe0856dc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.068+0000 7fe085edd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.068+0000 7fe085edd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:00.071 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.068+0000 7fe085edd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:00.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.068+0000 7fe0856dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:00.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.068+0000 7fe0856dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:00.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.068+0000 7fe0856dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:00.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.069+0000 7fe085edd640 10 monclient(hunting): handle_auth_done global_id 17768 payload 274 2023-12-15T17:24:00.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.069+0000 7fe085edd640 1 -- 172.21.15.69:0/1703829824 >> v2:172.21.15.192:3300/0 conn(0x7fe0800a8e10 msgr2=0x7fe080142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:00.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.069+0000 7fe085edd640 1 --2- 172.21.15.69:0/1703829824 >> v2:172.21.15.192:3300/0 conn(0x7fe0800a8e10 0x7fe080142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.072 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.069+0000 7fe085edd640 1 -- 172.21.15.69:0/1703829824 >> v2:172.21.15.69:3300/0 conn(0x7fe0801454a0 msgr2=0x7fe080145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:00.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.069+0000 7fe085edd640 1 --2- 172.21.15.69:0/1703829824 >> v2:172.21.15.69:3300/0 conn(0x7fe0801454a0 0x7fe080145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.069+0000 7fe085edd640 10 monclient: _finish_hunting 0 2023-12-15T17:24:00.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.069+0000 7fe085edd640 1 monclient: found mon.noname-b 2023-12-15T17:24:00.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.069+0000 7fe085edd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:00.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.069+0000 7fe085edd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:24:00.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.069+0000 7fe085edd640 1 -- 172.21.15.69:0/1703829824 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0800a97f0 con 0x7fe0800a8a30 2023-12-15T17:24:00.073 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.072+0000 7fe085edd640 1 --2- 172.21.15.69:0/1703829824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801424e0 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009870 tx=0x7fe07c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=673b32d5a221a459 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:00.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 1 -- 172.21.15.69:0/1703829824 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c031640 con 0x7fe0800a8a30 2023-12-15T17:24:00.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:00.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 10 monclient: dump: 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 10 monclient: _finish_auth 0 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 10 monclient: _check_auth_tickets 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:00.078 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 1 -- 172.21.15.69:0/1703829824 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c0317a0 con 0x7fe0800a8a30 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 1 -- 172.21.15.69:0/1703829824 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c031b30 con 0x7fe0800a8a30 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe084edb640 10 monclient: dump: 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:00.079 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:00.080 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe088141640 5 monclient: authenticate success, global_id 17768 2023-12-15T17:24:00.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe088141640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:00.080 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.076+0000 7fe088141640 4 set_mon_vals no callback set 2023-12-15T17:24:00.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 10 monclient: shutdown 2023-12-15T17:24:00.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:00.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 1 -- 172.21.15.69:0/1703829824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 msgr2=0x7fe0801424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:00.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 1 --2- 172.21.15.69:0/1703829824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801424e0 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009870 tx=0x7fe07c02b8f0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 1 -- 172.21.15.69:0/1703829824 shutdown_connections 2023-12-15T17:24:00.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 1 --2- 172.21.15.69:0/1703829824 >> v2:172.21.15.69:3300/0 conn(0x7fe0801454a0 0x7fe080145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 1 --2- 172.21.15.69:0/1703829824 >> v2:172.21.15.192:3300/0 conn(0x7fe0800a8e10 0x7fe080142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 1 --2- 172.21.15.69:0/1703829824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801424e0 unknown :-1 s=CLOSED pgs=3126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 1 -- 172.21.15.69:0/1703829824 >> 172.21.15.69:0/1703829824 conn(0x7fe080061a30 msgr2=0x7fe0800b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 1 -- 172.21.15.69:0/1703829824 shutdown_connections 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 1 -- 172.21.15.69:0/1703829824 wait complete. 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 10 monclient: build_initial_monmap 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.077+0000 7fe088141640 10 monclient: monmap: 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.084 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 1 librados: starting msgr at 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 1 librados: starting objecter 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 1 Processor -- start 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 1 -- start start 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 1 librados: setting wanted keys 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 1 librados: calling monclient init 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 10 monclient: init 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801d71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 10 monclient(hunting): picked mon.noname-c con 0x7fe0800a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.078+0000 7fe088141640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0800a8e10 0x7fe08013b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe088141640 10 monclient(hunting): picked mon.noname-b con 0x7fe0800a8e10 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe088141640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0801454a0 0x7fe08013bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe088141640 10 monclient(hunting): picked mon.noname-a con 0x7fe0801454a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe088141640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe088141640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800ae970 con 0x7fe0801454a0 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe088141640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe088141640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08013c320 con 0x7fe0800a8e10 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe088141640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe088141640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800b0d30 con 0x7fe0800a8a30 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe088141640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe088141640 10 monclient(hunting): authenticate will time out at 2744.395406s 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe085edd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801d71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:00.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe0866de640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0801454a0 0x7fe08013bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe0856dc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0800a8e10 0x7fe08013b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe085edd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801d71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42260/0 (socket says 172.21.15.69:42260) 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe085edd640 1 -- 172.21.15.69:0/3474233654 learned_addr learned my addr 172.21.15.69:0/3474233654 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe085edd640 10 monclient(hunting): get_auth_request con 0x7fe0800a8a30 auth_method 0 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe085edd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe085edd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe085edd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe0866de640 10 monclient(hunting): get_auth_request con 0x7fe0801454a0 auth_method 0 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe0866de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe0866de640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe0866de640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe0856dc640 10 monclient(hunting): get_auth_request con 0x7fe0800a8e10 auth_method 0 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe0856dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:00.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe0856dc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:00.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.079+0000 7fe0856dc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:00.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.080+0000 7fe085edd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:00.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.080+0000 7fe085edd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:00.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.080+0000 7fe085edd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:00.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.080+0000 7fe0866de640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:00.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.080+0000 7fe0866de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:00.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.080+0000 7fe0866de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:00.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.080+0000 7fe0856dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:00.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.080+0000 7fe0856dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:00.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.080+0000 7fe0856dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:00.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.082+0000 7fe085edd640 10 monclient(hunting): handle_auth_done global_id 17771 payload 824 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.082+0000 7fe085edd640 1 -- 172.21.15.69:0/3474233654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0800a8e10 msgr2=0x7fe08013b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.082+0000 7fe085edd640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0800a8e10 0x7fe08013b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.082+0000 7fe085edd640 1 -- 172.21.15.69:0/3474233654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0801454a0 msgr2=0x7fe08013bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.082+0000 7fe085edd640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0801454a0 0x7fe08013bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.082+0000 7fe085edd640 10 monclient: _finish_hunting 0 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.082+0000 7fe085edd640 1 monclient: found mon.noname-c 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.082+0000 7fe085edd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.082+0000 7fe085edd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.082+0000 7fe085edd640 1 -- 172.21.15.69:0/3474233654 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c0094d0 con 0x7fe0800a8a30 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.085+0000 7fe085edd640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801d71e0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7fe07c000c00 tx=0x7fe07c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.085+0000 7fe0767fc640 10 client.?.objecter ms_handle_connect 0x7fe0800a8a30 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.085+0000 7fe0767fc640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:00.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 1 -- 172.21.15.69:0/3474233654 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c03a930 con 0x7fe0800a8a30 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 10 monclient: dump: 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 10 monclient: _finish_auth 0 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 10 monclient: _check_auth_tickets 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 1 -- 172.21.15.69:0/3474233654 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c03aa90 con 0x7fe0800a8a30 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 1 -- 172.21.15.69:0/3474233654 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c03ae40 con 0x7fe0800a8a30 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe0767fc640 10 monclient: dump: 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.092 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe088141640 5 monclient: authenticate success, global_id 17771 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe088141640 10 monclient: _renew_subs 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe077fff640 4 set_mon_vals no callback set 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe088141640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe088141640 1 -- 172.21.15.69:0/3474233654 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0800adb70 con 0x7fe0800a8a30 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe088141640 10 client.17771.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe088141640 10 monclient: _renew_subs 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe088141640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe088141640 1 -- 172.21.15.69:0/3474233654 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0801d7c00 con 0x7fe0800a8a30 2023-12-15T17:24:00.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.088+0000 7fe088141640 1 librados: init done 2023-12-15T17:24:00.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.089+0000 7fe088141640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:00.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.089+0000 7fe088141640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:00.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.089+0000 7fe088141640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:00.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.089+0000 7fe088141640 1 -- 172.21.15.69:0/3474233654 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe080067ac0 con 0x7fe0800a8a30 2023-12-15T17:24:00.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.109+0000 7fe0767fc640 1 -- 172.21.15.69:0/3474233654 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe07c02f050 con 0x7fe0800a8a30 2023-12-15T17:24:00.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.109+0000 7fe0767fc640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe06406cba0 0x7fe06406f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.109+0000 7fe0856dc640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe06406cba0 0x7fe06406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:00.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.109+0000 7fe0856dc640 10 monclient: get_auth_request con 0x7fe06406cba0 auth_method 0 2023-12-15T17:24:00.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.110+0000 7fe0767fc640 10 client.17771.objecter ms_handle_connect 0x7fe06406cba0 2023-12-15T17:24:00.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.110+0000 7fe0856dc640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe06406cba0 0x7fe06406f060 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe0801db570 tx=0x7fe070009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:00.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.346+0000 7fe0767fc640 1 -- 172.21.15.69:0/3474233654 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(908..908 src has 534..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7fe07c07a600 con 0x7fe0800a8a30 2023-12-15T17:24:00.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.346+0000 7fe0767fc640 10 client.17771.objecter ms_dispatch 0x7fe080000b90 osd_map(908..908 src has 534..908) v4 2023-12-15T17:24:00.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.346+0000 7fe0767fc640 3 client.17771.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:00.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.346+0000 7fe0767fc640 3 client.17771.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:00.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.346+0000 7fe0767fc640 20 client.17771.objecter dump_active .. 0 homeless 2023-12-15T17:24:00.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.346+0000 7fe0767fc640 1 -- 172.21.15.69:0/3474233654 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe07c076940 con 0x7fe0800a8a30 2023-12-15T17:24:00.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.346+0000 7fe0767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:00.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.346+0000 7fe0767fc640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:00.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.532+0000 7fe088141640 1 -- 172.21.15.69:0/3474233654 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe080145e70 con 0x7fe06406cba0 2023-12-15T17:24:00.538 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.537+0000 7fe0767fc640 1 -- 172.21.15.69:0/3474233654 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151952 (secure 0 0 0) 0x7fe070011a30 con 0x7fe06406cba0 2023-12-15T17:24:00.540 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:24:00.541 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-12-15T17:24:00.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 10 librados: watch_flush enter 2023-12-15T17:24:00.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 10 librados: watch_flush exit 2023-12-15T17:24:00.542 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 20 client.17771.objecter shutdown clearing up homeless session... 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 10 client.17771.objecter successfully canceled tick 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 -- 172.21.15.69:0/3474233654 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe06406cba0 msgr2=0x7fe06406f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe06406cba0 0x7fe06406f060 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe0801db570 tx=0x7fe070009190 comp rx=0 tx=0).stop 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 10 monclient: shutdown 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 -- 172.21.15.69:0/3474233654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 msgr2=0x7fe0801d71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801d71e0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7fe07c000c00 tx=0x7fe07c0046c0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 -- 172.21.15.69:0/3474233654 shutdown_connections 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fe06406cba0 0x7fe06406f060 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0801454a0 0x7fe08013bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0800a8e10 0x7fe08013b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 --2- 172.21.15.69:0/3474233654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0800a8a30 0x7fe0801d71e0 unknown :-1 s=CLOSED pgs=3127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 -- 172.21.15.69:0/3474233654 >> 172.21.15.69:0/3474233654 conn(0x7fe080061a30 msgr2=0x7fe08013e340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:00.543 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 -- 172.21.15.69:0/3474233654 shutdown_connections 2023-12-15T17:24:00.544 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 -- 172.21.15.69:0/3474233654 wait complete. 2023-12-15T17:24:00.544 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.540+0000 7fe088141640 1 librados: shutdown 2023-12-15T17:24:00.552 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1579,"stamp":"2023-12-15T17:23:59.782162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":570600,"kb_used_data":44892,"kb_used_omap":12,"kb_used_meta":525555,"kb_avail":754404120,"statfs":{"total":773094113280,"available":772509818880,"internally_reserved":0,"allocated":45969408,"data_stored":24753545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":538168880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":394,"apply_latency_ms":394,"commit_latency_ns":394000000,"apply_latency_ns":394000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000712"},"pg_stats":[{"pgid":"7.1b","version":"0'0","reported_seq":869,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.257675+0000","last_change":"2023-12-15T17:03:26.413418+0000","last_active":"2023-12-15T17:23:57.257675+0000","last_peered":"2023-12-15T17:23:57.257675+0000","last_clean":"2023-12-15T17:23:57.257675+0000","last_became_active":"2023-12-15T17:03:26.401879+0000","last_became_peered":"2023-12-15T17:03:26.401879+0000","last_unstale":"2023-12-15T17:23:57.257675+0000","last_undegraded":"2023-12-15T17:23:57.257675+0000","last_fullsized":"2023-12-15T17:23:57.257675+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:25:02.124791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.128322+0000","last_change":"2023-12-15T17:03:43.168165+0000","last_active":"2023-12-15T17:23:56.128322+0000","last_peered":"2023-12-15T17:23:56.128322+0000","last_clean":"2023-12-15T17:23:56.128322+0000","last_became_active":"2023-12-15T17:03:43.159317+0000","last_became_peered":"2023-12-15T17:03:43.159317+0000","last_unstale":"2023-12-15T17:23:56.128322+0000","last_undegraded":"2023-12-15T17:23:56.128322+0000","last_fullsized":"2023-12-15T17:23:56.128322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:28:01.151029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.123155+0000","last_change":"2023-12-15T17:03:27.146666+0000","last_active":"2023-12-15T17:23:56.123155+0000","last_peered":"2023-12-15T17:23:56.123155+0000","last_clean":"2023-12-15T17:23:56.123155+0000","last_became_active":"2023-12-15T17:03:27.137993+0000","last_became_peered":"2023-12-15T17:03:27.137993+0000","last_unstale":"2023-12-15T17:23:56.123155+0000","last_undegraded":"2023-12-15T17:23:56.123155+0000","last_fullsized":"2023-12-15T17:23:56.123155+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:13:48.832295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":869,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.238398+0000","last_change":"2023-12-15T17:03:32.164840+0000","last_active":"2023-12-15T17:23:57.238398+0000","last_peered":"2023-12-15T17:23:57.238398+0000","last_clean":"2023-12-15T17:23:57.238398+0000","last_became_active":"2023-12-15T17:03:32.155432+0000","last_became_peered":"2023-12-15T17:03:32.155432+0000","last_unstale":"2023-12-15T17:23:57.238398+0000","last_undegraded":"2023-12-15T17:23:57.238398+0000","last_fullsized":"2023-12-15T17:23:57.238398+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:29:00.934002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":845,"reported_epoch":904,"state":"active+clean","last_fresh":"2023-12-15T17:23:55.320927+0000","last_change":"2023-12-15T17:03:32.049516+0000","last_active":"2023-12-15T17:23:55.320927+0000","last_peered":"2023-12-15T17:23:55.320927+0000","last_clean":"2023-12-15T17:23:55.320927+0000","last_became_active":"2023-12-15T17:03:32.040162+0000","last_became_peered":"2023-12-15T17:03:32.040162+0000","last_unstale":"2023-12-15T17:23:55.320927+0000","last_undegraded":"2023-12-15T17:23:55.320927+0000","last_fullsized":"2023-12-15T17:23:55.320927+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:24:16.619046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":854,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.908823+0000","last_change":"2023-12-15T17:03:29.590131+0000","last_active":"2023-12-15T17:23:56.908823+0000","last_peered":"2023-12-15T17:23:56.908823+0000","last_clean":"2023-12-15T17:23:56.908823+0000","last_became_active":"2023-12-15T17:03:29.584729+0000","last_became_peered":"2023-12-15T17:03:29.584729+0000","last_unstale":"2023-12-15T17:23:56.908823+0000","last_undegraded":"2023-12-15T17:23:56.908823+0000","last_fullsized":"2023-12-15T17:23:56.908823+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:25:55.347921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":873,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.180768+0000","last_change":"2023-12-15T17:03:27.355790+0000","last_active":"2023-12-15T17:23:58.180768+0000","last_peered":"2023-12-15T17:23:58.180768+0000","last_clean":"2023-12-15T17:23:58.180768+0000","last_became_active":"2023-12-15T17:03:27.349390+0000","last_became_peered":"2023-12-15T17:03:27.349390+0000","last_unstale":"2023-12-15T17:23:58.180768+0000","last_undegraded":"2023-12-15T17:23:58.180768+0000","last_fullsized":"2023-12-15T17:23:58.180768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:50:15.500629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":849,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.754893+0000","last_change":"2023-12-15T17:03:41.190972+0000","last_active":"2023-12-15T17:23:56.754893+0000","last_peered":"2023-12-15T17:23:56.754893+0000","last_clean":"2023-12-15T17:23:56.754893+0000","last_became_active":"2023-12-15T17:03:41.182883+0000","last_became_peered":"2023-12-15T17:03:41.182883+0000","last_unstale":"2023-12-15T17:23:56.754893+0000","last_undegraded":"2023-12-15T17:23:56.754893+0000","last_fullsized":"2023-12-15T17:23:56.754893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T00:04:20.611993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":852,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.809588+0000","last_change":"2023-12-15T17:03:27.568351+0000","last_active":"2023-12-15T17:23:56.809588+0000","last_peered":"2023-12-15T17:23:56.809588+0000","last_clean":"2023-12-15T17:23:56.809588+0000","last_became_active":"2023-12-15T17:03:27.563090+0000","last_became_peered":"2023-12-15T17:03:27.563090+0000","last_unstale":"2023-12-15T17:23:56.809588+0000","last_undegraded":"2023-12-15T17:23:56.809588+0000","last_fullsized":"2023-12-15T17:23:56.809588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:48:11.443821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":847,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.656834+0000","last_change":"2023-12-15T17:03:26.203421+0000","last_active":"2023-12-15T17:23:56.656834+0000","last_peered":"2023-12-15T17:23:56.656834+0000","last_clean":"2023-12-15T17:23:56.656834+0000","last_became_active":"2023-12-15T17:03:26.168812+0000","last_became_peered":"2023-12-15T17:03:26.168812+0000","last_unstale":"2023-12-15T17:23:56.656834+0000","last_undegraded":"2023-12-15T17:23:56.656834+0000","last_fullsized":"2023-12-15T17:23:56.656834+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:30:33.793828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":849,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.744754+0000","last_change":"2023-12-15T17:03:33.666043+0000","last_active":"2023-12-15T17:23:56.744754+0000","last_peered":"2023-12-15T17:23:56.744754+0000","last_clean":"2023-12-15T17:23:56.744754+0000","last_became_active":"2023-12-15T17:03:33.660278+0000","last_became_peered":"2023-12-15T17:03:33.660278+0000","last_unstale":"2023-12-15T17:23:56.744754+0000","last_undegraded":"2023-12-15T17:23:56.744754+0000","last_fullsized":"2023-12-15T17:23:56.744754+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:21:15.535555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":852,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.757186+0000","last_change":"2023-12-15T17:03:29.499597+0000","last_active":"2023-12-15T17:23:56.757186+0000","last_peered":"2023-12-15T17:23:56.757186+0000","last_clean":"2023-12-15T17:23:56.757186+0000","last_became_active":"2023-12-15T17:03:29.479817+0000","last_became_peered":"2023-12-15T17:03:29.479817+0000","last_unstale":"2023-12-15T17:23:56.757186+0000","last_undegraded":"2023-12-15T17:23:56.757186+0000","last_fullsized":"2023-12-15T17:23:56.757186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:29:30.677748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":870,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.247428+0000","last_change":"2023-12-15T17:03:27.798402+0000","last_active":"2023-12-15T17:23:57.247428+0000","last_peered":"2023-12-15T17:23:57.247428+0000","last_clean":"2023-12-15T17:23:57.247428+0000","last_became_active":"2023-12-15T17:03:27.764430+0000","last_became_peered":"2023-12-15T17:03:27.764430+0000","last_unstale":"2023-12-15T17:23:57.247428+0000","last_undegraded":"2023-12-15T17:23:57.247428+0000","last_fullsized":"2023-12-15T17:23:57.247428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:50:51.103963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":882,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.172032+0000","last_change":"2023-12-15T17:02:01.284924+0000","last_active":"2023-12-15T17:23:56.172032+0000","last_peered":"2023-12-15T17:23:56.172032+0000","last_clean":"2023-12-15T17:23:56.172032+0000","last_became_active":"2023-12-15T17:02:01.275938+0000","last_became_peered":"2023-12-15T17:02:01.275938+0000","last_unstale":"2023-12-15T17:23:56.172032+0000","last_undegraded":"2023-12-15T17:23:56.172032+0000","last_fullsized":"2023-12-15T17:23:56.172032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:11:11.381089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":874,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.177122+0000","last_change":"2023-12-15T17:03:29.741478+0000","last_active":"2023-12-15T17:23:58.177122+0000","last_peered":"2023-12-15T17:23:58.177122+0000","last_clean":"2023-12-15T17:23:58.177122+0000","last_became_active":"2023-12-15T17:03:29.736171+0000","last_became_peered":"2023-12-15T17:03:29.736171+0000","last_unstale":"2023-12-15T17:23:58.177122+0000","last_undegraded":"2023-12-15T17:23:58.177122+0000","last_fullsized":"2023-12-15T17:23:58.177122+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:42:10.131036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":876,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.267646+0000","last_change":"2023-12-15T17:01:59.949551+0000","last_active":"2023-12-15T17:23:57.267646+0000","last_peered":"2023-12-15T17:23:57.267646+0000","last_clean":"2023-12-15T17:23:57.267646+0000","last_became_active":"2023-12-15T17:01:59.918026+0000","last_became_peered":"2023-12-15T17:01:59.918026+0000","last_unstale":"2023-12-15T17:23:57.267646+0000","last_undegraded":"2023-12-15T17:23:57.267646+0000","last_fullsized":"2023-12-15T17:23:57.267646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:10:51.112278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":849,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.663013+0000","last_change":"2023-12-15T17:03:26.256693+0000","last_active":"2023-12-15T17:23:56.663013+0000","last_peered":"2023-12-15T17:23:56.663013+0000","last_clean":"2023-12-15T17:23:56.663013+0000","last_became_active":"2023-12-15T17:03:26.235756+0000","last_became_peered":"2023-12-15T17:03:26.235756+0000","last_unstale":"2023-12-15T17:23:56.663013+0000","last_undegraded":"2023-12-15T17:23:56.663013+0000","last_fullsized":"2023-12-15T17:23:56.663013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:08:19.767760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":882,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.147791+0000","last_change":"2023-12-15T17:02:01.380902+0000","last_active":"2023-12-15T17:23:56.147791+0000","last_peered":"2023-12-15T17:23:56.147791+0000","last_clean":"2023-12-15T17:23:56.147791+0000","last_became_active":"2023-12-15T17:02:01.375672+0000","last_became_peered":"2023-12-15T17:02:01.375672+0000","last_unstale":"2023-12-15T17:23:56.147791+0000","last_undegraded":"2023-12-15T17:23:56.147791+0000","last_fullsized":"2023-12-15T17:23:56.147791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:15:17.260715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"32'7","reported_seq":863,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.747693+0000","last_change":"2023-12-15T17:03:26.359500+0000","last_active":"2023-12-15T17:23:56.747693+0000","last_peered":"2023-12-15T17:23:56.747693+0000","last_clean":"2023-12-15T17:23:56.747693+0000","last_became_active":"2023-12-15T17:03:26.343248+0000","last_became_peered":"2023-12-15T17:03:26.343248+0000","last_unstale":"2023-12-15T17:23:56.747693+0000","last_undegraded":"2023-12-15T17:23:56.747693+0000","last_fullsized":"2023-12-15T17:23:56.747693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:31:17.788444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":881,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.185109+0000","last_change":"2023-12-15T17:01:59.938931+0000","last_active":"2023-12-15T17:23:58.185109+0000","last_peered":"2023-12-15T17:23:58.185109+0000","last_clean":"2023-12-15T17:23:58.185109+0000","last_became_active":"2023-12-15T17:01:59.908680+0000","last_became_peered":"2023-12-15T17:01:59.908680+0000","last_unstale":"2023-12-15T17:23:58.185109+0000","last_undegraded":"2023-12-15T17:23:58.185109+0000","last_fullsized":"2023-12-15T17:23:58.185109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:35:37.366491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.170131+0000","last_change":"2023-12-15T17:03:27.788302+0000","last_active":"2023-12-15T17:23:56.170131+0000","last_peered":"2023-12-15T17:23:56.170131+0000","last_clean":"2023-12-15T17:23:56.170131+0000","last_became_active":"2023-12-15T17:03:27.780434+0000","last_became_peered":"2023-12-15T17:03:27.780434+0000","last_unstale":"2023-12-15T17:23:56.170131+0000","last_undegraded":"2023-12-15T17:23:56.170131+0000","last_fullsized":"2023-12-15T17:23:56.170131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:24:22.326668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":883,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.126527+0000","last_change":"2023-12-15T17:02:01.718405+0000","last_active":"2023-12-15T17:23:56.126527+0000","last_peered":"2023-12-15T17:23:56.126527+0000","last_clean":"2023-12-15T17:23:56.126527+0000","last_became_active":"2023-12-15T17:02:01.713188+0000","last_became_peered":"2023-12-15T17:02:01.713188+0000","last_unstale":"2023-12-15T17:23:56.126527+0000","last_undegraded":"2023-12-15T17:23:56.126527+0000","last_fullsized":"2023-12-15T17:23:56.126527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:27:19.133287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":870,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.253473+0000","last_change":"2023-12-15T17:03:26.257800+0000","last_active":"2023-12-15T17:23:57.253473+0000","last_peered":"2023-12-15T17:23:57.253473+0000","last_clean":"2023-12-15T17:23:57.253473+0000","last_became_active":"2023-12-15T17:03:26.252650+0000","last_became_peered":"2023-12-15T17:03:26.252650+0000","last_unstale":"2023-12-15T17:23:57.253473+0000","last_undegraded":"2023-12-15T17:23:57.253473+0000","last_fullsized":"2023-12-15T17:23:57.253473+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:20:58.993074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":859,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.780879+0000","last_change":"2023-12-15T17:01:59.559261+0000","last_active":"2023-12-15T17:23:56.780879+0000","last_peered":"2023-12-15T17:23:56.780879+0000","last_clean":"2023-12-15T17:23:56.780879+0000","last_became_active":"2023-12-15T17:01:59.531480+0000","last_became_peered":"2023-12-15T17:01:59.531480+0000","last_unstale":"2023-12-15T17:23:56.780879+0000","last_undegraded":"2023-12-15T17:23:56.780879+0000","last_fullsized":"2023-12-15T17:23:56.780879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:51:07.527384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":876,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.146161+0000","last_change":"2023-12-15T17:03:29.375364+0000","last_active":"2023-12-15T17:23:56.146161+0000","last_peered":"2023-12-15T17:23:56.146161+0000","last_clean":"2023-12-15T17:23:56.146161+0000","last_became_active":"2023-12-15T17:03:29.361304+0000","last_became_peered":"2023-12-15T17:03:29.361304+0000","last_unstale":"2023-12-15T17:23:56.146161+0000","last_undegraded":"2023-12-15T17:23:56.146161+0000","last_fullsized":"2023-12-15T17:23:56.146161+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:51:08.058570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":882,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.102856+0000","last_change":"2023-12-15T17:02:01.240306+0000","last_active":"2023-12-15T17:23:56.102856+0000","last_peered":"2023-12-15T17:23:56.102856+0000","last_clean":"2023-12-15T17:23:56.102856+0000","last_became_active":"2023-12-15T17:02:01.208353+0000","last_became_peered":"2023-12-15T17:02:01.208353+0000","last_unstale":"2023-12-15T17:23:56.102856+0000","last_undegraded":"2023-12-15T17:23:56.102856+0000","last_fullsized":"2023-12-15T17:23:56.102856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:41:20.461821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":874,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.191435+0000","last_change":"2023-12-15T17:03:27.328921+0000","last_active":"2023-12-15T17:23:58.191435+0000","last_peered":"2023-12-15T17:23:58.191435+0000","last_clean":"2023-12-15T17:23:58.191435+0000","last_became_active":"2023-12-15T17:03:27.321338+0000","last_became_peered":"2023-12-15T17:03:27.321338+0000","last_unstale":"2023-12-15T17:23:58.191435+0000","last_undegraded":"2023-12-15T17:23:58.191435+0000","last_fullsized":"2023-12-15T17:23:58.191435+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:51:42.546700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":883,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.120609+0000","last_change":"2023-12-15T17:02:01.625426+0000","last_active":"2023-12-15T17:23:56.120609+0000","last_peered":"2023-12-15T17:23:56.120609+0000","last_clean":"2023-12-15T17:23:56.120609+0000","last_became_active":"2023-12-15T17:02:01.616517+0000","last_became_peered":"2023-12-15T17:02:01.616517+0000","last_unstale":"2023-12-15T17:23:56.120609+0000","last_undegraded":"2023-12-15T17:23:56.120609+0000","last_fullsized":"2023-12-15T17:23:56.120609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:54:17.323141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":961,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.661499+0000","last_change":"2023-12-15T17:01:52.907228+0000","last_active":"2023-12-15T17:23:56.661499+0000","last_peered":"2023-12-15T17:23:56.661499+0000","last_clean":"2023-12-15T17:23:56.661499+0000","last_became_active":"2023-12-15T17:01:52.878062+0000","last_became_peered":"2023-12-15T17:01:52.878062+0000","last_unstale":"2023-12-15T17:23:56.661499+0000","last_undegraded":"2023-12-15T17:23:56.661499+0000","last_fullsized":"2023-12-15T17:23:56.661499+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_clean_scrub_stamp":"2023-12-15T17:01:50.252516+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:47:14.717362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":850,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.753308+0000","last_change":"2023-12-15T17:03:31.083758+0000","last_active":"2023-12-15T17:23:56.753308+0000","last_peered":"2023-12-15T17:23:56.753308+0000","last_clean":"2023-12-15T17:23:56.753308+0000","last_became_active":"2023-12-15T17:03:31.075388+0000","last_became_peered":"2023-12-15T17:03:31.075388+0000","last_unstale":"2023-12-15T17:23:56.753308+0000","last_undegraded":"2023-12-15T17:23:56.753308+0000","last_fullsized":"2023-12-15T17:23:56.753308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:32:49.564294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":848,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.697108+0000","last_change":"2023-12-15T17:03:29.701214+0000","last_active":"2023-12-15T17:23:56.697108+0000","last_peered":"2023-12-15T17:23:56.697108+0000","last_clean":"2023-12-15T17:23:56.697108+0000","last_became_active":"2023-12-15T17:03:29.683464+0000","last_became_peered":"2023-12-15T17:03:29.683464+0000","last_unstale":"2023-12-15T17:23:56.697108+0000","last_undegraded":"2023-12-15T17:23:56.697108+0000","last_fullsized":"2023-12-15T17:23:56.697108+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:02:06.165049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":853,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.906574+0000","last_change":"2023-12-15T17:03:26.043197+0000","last_active":"2023-12-15T17:23:56.906574+0000","last_peered":"2023-12-15T17:23:56.906574+0000","last_clean":"2023-12-15T17:23:56.906574+0000","last_became_active":"2023-12-15T17:03:26.036270+0000","last_became_peered":"2023-12-15T17:03:26.036270+0000","last_unstale":"2023-12-15T17:23:56.906574+0000","last_undegraded":"2023-12-15T17:23:56.906574+0000","last_fullsized":"2023-12-15T17:23:56.906574+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:27:28.632284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":851,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.960741+0000","last_change":"2023-12-15T17:03:29.615007+0000","last_active":"2023-12-15T17:23:56.960741+0000","last_peered":"2023-12-15T17:23:56.960741+0000","last_clean":"2023-12-15T17:23:56.960741+0000","last_became_active":"2023-12-15T17:03:29.608805+0000","last_became_peered":"2023-12-15T17:03:29.608805+0000","last_unstale":"2023-12-15T17:23:56.960741+0000","last_undegraded":"2023-12-15T17:23:56.960741+0000","last_fullsized":"2023-12-15T17:23:56.960741+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T20:35:50.230477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":875,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.167794+0000","last_change":"2023-12-15T17:03:27.507868+0000","last_active":"2023-12-15T17:23:58.167794+0000","last_peered":"2023-12-15T17:23:58.167794+0000","last_clean":"2023-12-15T17:23:58.167794+0000","last_became_active":"2023-12-15T17:03:27.499200+0000","last_became_peered":"2023-12-15T17:03:27.499200+0000","last_unstale":"2023-12-15T17:23:58.167794+0000","last_undegraded":"2023-12-15T17:23:58.167794+0000","last_fullsized":"2023-12-15T17:23:58.167794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T20:31:29.233160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":875,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.163182+0000","last_change":"2023-12-15T17:03:28.022217+0000","last_active":"2023-12-15T17:23:58.163182+0000","last_peered":"2023-12-15T17:23:58.163182+0000","last_clean":"2023-12-15T17:23:58.163182+0000","last_became_active":"2023-12-15T17:03:28.012736+0000","last_became_peered":"2023-12-15T17:03:28.012736+0000","last_unstale":"2023-12-15T17:23:58.163182+0000","last_undegraded":"2023-12-15T17:23:58.163182+0000","last_fullsized":"2023-12-15T17:23:58.163182+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T23:37:05.516314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":870,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.245793+0000","last_change":"2023-12-15T17:03:27.802217+0000","last_active":"2023-12-15T17:23:57.245793+0000","last_peered":"2023-12-15T17:23:57.245793+0000","last_clean":"2023-12-15T17:23:57.245793+0000","last_became_active":"2023-12-15T17:03:27.791224+0000","last_became_peered":"2023-12-15T17:03:27.791224+0000","last_unstale":"2023-12-15T17:23:57.245793+0000","last_undegraded":"2023-12-15T17:23:57.245793+0000","last_fullsized":"2023-12-15T17:23:57.245793+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:54:03.758840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":851,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.326424+0000","last_change":"2023-12-15T17:03:34.018076+0000","last_active":"2023-12-15T17:23:58.326424+0000","last_peered":"2023-12-15T17:23:58.326424+0000","last_clean":"2023-12-15T17:23:58.326424+0000","last_became_active":"2023-12-15T17:03:33.897262+0000","last_became_peered":"2023-12-15T17:03:33.897262+0000","last_unstale":"2023-12-15T17:23:58.326424+0000","last_undegraded":"2023-12-15T17:23:58.326424+0000","last_fullsized":"2023-12-15T17:23:58.326424+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T23:33:26.034906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":874,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.183460+0000","last_change":"2023-12-15T17:03:28.235706+0000","last_active":"2023-12-15T17:23:58.183460+0000","last_peered":"2023-12-15T17:23:58.183460+0000","last_clean":"2023-12-15T17:23:58.183460+0000","last_became_active":"2023-12-15T17:03:28.164968+0000","last_became_peered":"2023-12-15T17:03:28.164968+0000","last_unstale":"2023-12-15T17:23:58.183460+0000","last_undegraded":"2023-12-15T17:23:58.183460+0000","last_fullsized":"2023-12-15T17:23:58.183460+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:55:52.108198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":875,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.143397+0000","last_change":"2023-12-15T17:03:29.256120+0000","last_active":"2023-12-15T17:23:56.143397+0000","last_peered":"2023-12-15T17:23:56.143397+0000","last_clean":"2023-12-15T17:23:56.143397+0000","last_became_active":"2023-12-15T17:03:29.219241+0000","last_became_peered":"2023-12-15T17:03:29.219241+0000","last_unstale":"2023-12-15T17:23:56.143397+0000","last_undegraded":"2023-12-15T17:23:56.143397+0000","last_fullsized":"2023-12-15T17:23:56.143397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:27:29.782496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":847,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.691386+0000","last_change":"2023-12-15T17:03:25.431259+0000","last_active":"2023-12-15T17:23:56.691386+0000","last_peered":"2023-12-15T17:23:56.691386+0000","last_clean":"2023-12-15T17:23:56.691386+0000","last_became_active":"2023-12-15T17:03:25.394333+0000","last_became_peered":"2023-12-15T17:03:25.394333+0000","last_unstale":"2023-12-15T17:23:56.691386+0000","last_undegraded":"2023-12-15T17:23:56.691386+0000","last_fullsized":"2023-12-15T17:23:56.691386+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:35:32.135097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":847,"reported_epoch":905,"state":"active+clean","last_fresh":"2023-12-15T17:23:56.694198+0000","last_change":"2023-12-15T17:03:25.933659+0000","last_active":"2023-12-15T17:23:56.694198+0000","last_peered":"2023-12-15T17:23:56.694198+0000","last_clean":"2023-12-15T17:23:56.694198+0000","last_became_active":"2023-12-15T17:03:25.924138+0000","last_became_peered":"2023-12-15T17:03:25.924138+0000","last_unstale":"2023-12-15T17:23:56.694198+0000","last_undegraded":"2023-12-15T17:23:56.694198+0000","last_fullsized":"2023-12-15T17:23:56.694198+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:30:47.546645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509707,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58908,"kb_used_data":5512,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312932,"statfs":{"total":96636764160,"available":96576442368,"internally_reserved":0,"allocated":5644288,"data_stored":2992122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":10.289999999999999,"5min":13.026999999999999,"15min":16.065000000000001},"min":{"1min":5.0090000000000003,"5min":4.351,"15min":4.3390000000000004},"max":{"1min":20.521000000000001,"5min":151.61099999999999,"15min":199.32400000000001},"last":17.937000000000001},{"interface":"front","average":{"1min":10.064,"5min":14.347,"15min":17.850999999999999},"min":{"1min":6.2839999999999998,"5min":5.0149999999999997,"15min":4.9370000000000003},"max":{"1min":20.202999999999999,"5min":278.32799999999997,"15min":330.613},"last":21.178000000000001}]},{"osd":1,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":16.135000000000002,"5min":17.571999999999999,"15min":19.690000000000001},"min":{"1min":7.4400000000000004,"5min":6.7439999999999998,"15min":5.726},"max":{"1min":65.194999999999993,"5min":159.22399999999999,"15min":303.95400000000001},"last":7.335},{"interface":"front","average":{"1min":15.039999999999999,"5min":19.803000000000001,"15min":20.053999999999998},"min":{"1min":7.5439999999999996,"5min":6.923,"15min":6.4610000000000003},"max":{"1min":55.744,"5min":326.125,"15min":326.125},"last":21.805}]},{"osd":3,"last update":"Fri Dec 15 17:23:02 2023","interfaces":[{"interface":"back","average":{"1min":10.766999999999999,"5min":17.425999999999998,"15min":23.780000000000001},"min":{"1min":6.6740000000000004,"5min":5.944,"15min":5.9050000000000002},"max":{"1min":30.068000000000001,"5min":309.05399999999997,"15min":313.50400000000002},"last":7.7539999999999996},{"interface":"front","average":{"1min":15.036,"5min":20.998999999999999,"15min":25.573},"min":{"1min":7.476,"5min":7.2480000000000002,"15min":6.1449999999999996},"max":{"1min":28.457999999999998,"5min":372.85199999999998,"15min":372.85199999999998},"last":20.539999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":15.255000000000001,"5min":23.908000000000001,"15min":24.216000000000001},"min":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":6.5359999999999996},"max":{"1min":26.652000000000001,"5min":397.04000000000002,"15min":397.04000000000002},"last":28.001000000000001},{"interface":"front","average":{"1min":16.533999999999999,"5min":24.268999999999998,"15min":25.311},"min":{"1min":8.5980000000000008,"5min":8.5980000000000008,"15min":7.7809999999999997},"max":{"1min":31.385999999999999,"5min":217.01599999999999,"15min":217.01599999999999},"last":16.015000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:08 2023","interfaces":[{"interface":"back","average":{"1min":17.076000000000001,"5min":22.579000000000001,"15min":34.334000000000003},"min":{"1min":8.2629999999999999,"5min":8.2629999999999999,"15min":7.7839999999999998},"max":{"1min":29.724,"5min":196.75700000000001,"15min":493.322},"last":17.254000000000001},{"interface":"front","average":{"1min":13.827,"5min":18.082000000000001,"15min":30.268000000000001},"min":{"1min":7.5880000000000001,"5min":5.9160000000000004,"15min":5.9160000000000004},"max":{"1min":25.835000000000001,"5min":280.45400000000001,"15min":497.67200000000003},"last":22.481999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:49 2023","interfaces":[{"interface":"back","average":{"1min":16.161000000000001,"5min":22.856000000000002,"15min":25.489999999999998},"min":{"1min":10.462,"5min":9.4039999999999999,"15min":8.5969999999999995},"max":{"1min":25.347999999999999,"5min":289.68400000000003,"15min":289.68400000000003},"last":36.061},{"interface":"front","average":{"1min":14.811999999999999,"5min":21.321999999999999,"15min":22.838999999999999},"min":{"1min":9.6989999999999998,"5min":7.8200000000000003,"15min":6.9409999999999998},"max":{"1min":22.036000000000001,"5min":274.80099999999999,"15min":274.80099999999999},"last":26.370000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:02 2023","interfaces":[{"interface":"back","average":{"1min":19.841999999999999,"5min":24.600999999999999,"15min":31.303000000000001},"min":{"1min":9.3930000000000007,"5min":9.3930000000000007,"15min":8.5820000000000007},"max":{"1min":32.054000000000002,"5min":278.98000000000002,"15min":613.21299999999997},"last":29.344999999999999},{"interface":"front","average":{"1min":17.178999999999998,"5min":22.213999999999999,"15min":29.68},"min":{"1min":7.8360000000000003,"5min":7.8360000000000003,"15min":7.8360000000000003},"max":{"1min":29.925999999999998,"5min":275.72300000000001,"15min":613.91099999999994},"last":23.212}]}]},{"osd":1,"up_from":15,"seq":64424509710,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81508,"kb_used_data":5904,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290332,"statfs":{"total":96636764160,"available":96553299968,"internally_reserved":0,"allocated":6045696,"data_stored":3389962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":12.414999999999999,"5min":16.533000000000001,"15min":22.98},"min":{"1min":5.2759999999999998,"5min":4.4870000000000001,"15min":4.383},"max":{"1min":36.951999999999998,"5min":94.881,"15min":452.79500000000002},"last":7.0010000000000003},{"interface":"front","average":{"1min":16.593,"5min":15.000999999999999,"15min":17.280000000000001},"min":{"1min":5.2960000000000003,"5min":5.0419999999999998,"15min":4.3449999999999998},"max":{"1min":90.009,"5min":127.498,"15min":196.59200000000001},"last":10.116}]},{"osd":2,"last update":"Fri Dec 15 17:23:03 2023","interfaces":[{"interface":"back","average":{"1min":24.509,"5min":19.314,"15min":25.472000000000001},"min":{"1min":6.4480000000000004,"5min":5.7380000000000004,"15min":5.5190000000000001},"max":{"1min":276.47000000000003,"5min":276.47000000000003,"15min":482.72800000000001},"last":18.846},{"interface":"front","average":{"1min":25.614000000000001,"5min":22.815000000000001,"15min":31.640999999999998},"min":{"1min":8.2119999999999997,"5min":6.5629999999999997,"15min":6.1699999999999999},"max":{"1min":261.74400000000003,"5min":261.74400000000003,"15min":454.017},"last":17.173999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":19.303000000000001,"5min":20.509,"15min":28.212},"min":{"1min":7.1100000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"max":{"1min":110.723,"5min":147.37100000000001,"15min":488.23700000000002},"last":33.832000000000001},{"interface":"front","average":{"1min":16.472000000000001,"5min":21.302,"15min":26.646000000000001},"min":{"1min":10.347,"5min":9.1549999999999994,"15min":6.4119999999999999},"max":{"1min":30.919,"5min":110.056,"15min":333.53199999999998},"last":28.303000000000001}]},{"osd":4,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":13.672000000000001,"5min":18.148,"15min":26.068999999999999},"min":{"1min":7.1959999999999997,"5min":6.2629999999999999,"15min":5.8319999999999999},"max":{"1min":53.085999999999999,"5min":143.24000000000001,"15min":438.75099999999998},"last":7.8970000000000002},{"interface":"front","average":{"1min":18.068999999999999,"5min":21.373000000000001,"15min":31.120000000000001},"min":{"1min":10.132999999999999,"5min":8.4510000000000005,"15min":8.4339999999999993},"max":{"1min":41.640000000000001,"5min":78.664000000000001,"15min":443.58300000000003},"last":27.532}]},{"osd":5,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":22.023,"5min":20.832000000000001,"15min":36.176000000000002},"min":{"1min":8.4710000000000001,"5min":6.8369999999999997,"15min":6.8369999999999997},"max":{"1min":112.105,"5min":113.04300000000001,"15min":451.70999999999998},"last":8.9260000000000002},{"interface":"front","average":{"1min":31.739999999999998,"5min":27.744,"15min":29.515999999999998},"min":{"1min":12.116,"5min":9.8840000000000003,"15min":6.6500000000000004},"max":{"1min":209.96600000000001,"5min":209.96600000000001,"15min":264.46899999999999},"last":31.350999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":21.939,"5min":22.827999999999999,"15min":31.187999999999999},"min":{"1min":12.606,"5min":8.6809999999999992,"15min":7.5629999999999997},"max":{"1min":90.926000000000002,"5min":162.06200000000001,"15min":455.43000000000001},"last":34.859000000000002},{"interface":"front","average":{"1min":19.638999999999999,"5min":21.096,"15min":28.334},"min":{"1min":9.1400000000000006,"5min":7.3399999999999999,"15min":6.7249999999999996},"max":{"1min":94.715999999999994,"5min":169.34,"15min":314.589},"last":18.175000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":21.369,"5min":26.395,"15min":36.750999999999998},"min":{"1min":10.898999999999999,"5min":10.691000000000001,"15min":10.561999999999999},"max":{"1min":59.499000000000002,"5min":132.423,"15min":453.04300000000001},"last":28.978000000000002},{"interface":"front","average":{"1min":20.079999999999998,"5min":21.163,"15min":28.721},"min":{"1min":10.175000000000001,"5min":10.175000000000001,"15min":8.6110000000000007},"max":{"1min":63.219999999999999,"5min":77.674000000000007,"15min":239.51499999999999},"last":35.472999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58892,"kb_used_data":5496,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312948,"statfs":{"total":96636764160,"available":96576458752,"internally_reserved":0,"allocated":5627904,"data_stored":2985572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":13.765000000000001,"5min":13.314,"15min":18.777000000000001},"min":{"1min":6.2359999999999998,"5min":4.3639999999999999,"15min":4.1429999999999998},"max":{"1min":80.552999999999997,"5min":80.552999999999997,"15min":375.77800000000002},"last":10.553000000000001},{"interface":"front","average":{"1min":17.832999999999998,"5min":17.524999999999999,"15min":23.969000000000001},"min":{"1min":6.0199999999999996,"5min":4.8170000000000002,"15min":4.8170000000000002},"max":{"1min":94.995999999999995,"5min":94.995999999999995,"15min":556.42700000000002},"last":7.6689999999999996}]},{"osd":2,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":20.041,"5min":19.704999999999998,"15min":21.385999999999999},"min":{"1min":9.3889999999999993,"5min":6.2939999999999996,"15min":6.2939999999999996},"max":{"1min":99.216999999999999,"5min":248.864,"15min":396.13400000000001},"last":9.2669999999999995},{"interface":"front","average":{"1min":16.388999999999999,"5min":21.504999999999999,"15min":25.138999999999999},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":5.9790000000000001},"max":{"1min":75.944000000000003,"5min":234.161,"15min":354.649},"last":11.680999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":19.158000000000001,"5min":21.491,"15min":22.245000000000001},"min":{"1min":7.6159999999999997,"5min":6.1740000000000004,"15min":6.0830000000000002},"max":{"1min":96.188000000000002,"5min":99.453000000000003,"15min":413.00599999999997},"last":93.516000000000005},{"interface":"front","average":{"1min":15.550000000000001,"5min":19.518999999999998,"15min":24.094999999999999},"min":{"1min":6.0640000000000001,"5min":6.04,"15min":6.04},"max":{"1min":101.57299999999999,"5min":106.357,"15min":526.98599999999999},"last":84.674000000000007}]},{"osd":4,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":21.193999999999999,"5min":24.096,"15min":24.780999999999999},"min":{"1min":7.1829999999999998,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":119.754,"5min":266.279,"15min":389.75700000000001},"last":7.1539999999999999},{"interface":"front","average":{"1min":17.59,"5min":22.911999999999999,"15min":25.393000000000001},"min":{"1min":8.4700000000000006,"5min":8.4700000000000006,"15min":7.3520000000000003},"max":{"1min":37.862000000000002,"5min":227.19200000000001,"15min":436.572},"last":12.255000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":17.655000000000001,"5min":22.420000000000002,"15min":26.187999999999999},"min":{"1min":7.8220000000000001,"5min":7.5359999999999996,"15min":7.3540000000000001},"max":{"1min":67.956999999999994,"5min":121.137,"15min":449.81299999999999},"last":224.11099999999999},{"interface":"front","average":{"1min":20.068000000000001,"5min":23.791,"15min":27.331},"min":{"1min":10.81,"5min":7.7439999999999998,"15min":6.2539999999999996},"max":{"1min":65.381,"5min":109.90000000000001,"15min":429.75299999999999},"last":227.60900000000001}]},{"osd":6,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":19.919,"5min":23.132999999999999,"15min":25.440999999999999},"min":{"1min":10.682,"5min":8.6760000000000002,"15min":7.4699999999999998},"max":{"1min":91.619,"5min":93.299000000000007,"15min":469.86799999999999},"last":15.897},{"interface":"front","average":{"1min":23.565000000000001,"5min":24.164000000000001,"15min":27.797999999999998},"min":{"1min":12.039,"5min":9.2899999999999991,"15min":8.4740000000000002},"max":{"1min":102.664,"5min":102.664,"15min":373.93599999999998},"last":13.752000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":17.504999999999999,"5min":21.597999999999999,"15min":29.437999999999999},"min":{"1min":8.3109999999999999,"5min":7.0449999999999999,"15min":7.0449999999999999},"max":{"1min":77.521000000000001,"5min":151.999,"15min":416.24400000000003},"last":9.9390000000000001},{"interface":"front","average":{"1min":24.347000000000001,"5min":26.122,"15min":27.091000000000001},"min":{"1min":11.058,"5min":9.8610000000000007,"15min":7.7160000000000002},"max":{"1min":100.611,"5min":163.09700000000001,"15min":410.04300000000001},"last":14.868}]}]},{"osd":3,"up_from":15,"seq":64424509706,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76944,"kb_used_data":5508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294896,"statfs":{"total":96636764160,"available":96557973504,"internally_reserved":0,"allocated":5640192,"data_stored":2992117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":8.1609999999999996,"5min":13.029,"15min":18.565999999999999},"min":{"1min":5.3899999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"max":{"1min":14.663,"5min":132.791,"15min":252.483},"last":4.7389999999999999},{"interface":"front","average":{"1min":8.7210000000000001,"5min":13.803000000000001,"15min":18.956},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":24.484999999999999,"5min":138.03800000000001,"15min":235.10300000000001},"last":7.641}]},{"osd":1,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":10.631,"5min":13.651,"15min":22},"min":{"1min":7.4589999999999996,"5min":4.9390000000000001,"15min":4.9390000000000001},"max":{"1min":19.66,"5min":79.147000000000006,"15min":624.94100000000003},"last":21.771999999999998},{"interface":"front","average":{"1min":14.163,"5min":17.541,"15min":24.814},"min":{"1min":8.8680000000000003,"5min":7.4770000000000003,"15min":6.9740000000000002},"max":{"1min":22.484000000000002,"5min":86.152000000000001,"15min":635.13199999999995},"last":32.859999999999999}]},{"osd":2,"last update":"Fri Dec 15 17:23:45 2023","interfaces":[{"interface":"back","average":{"1min":14.015000000000001,"5min":17.581,"15min":22.074999999999999},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":60.210000000000001,"5min":122.60899999999999,"15min":286.22699999999998},"last":10.489000000000001},{"interface":"front","average":{"1min":13.85,"5min":17.390000000000001,"15min":25.114000000000001},"min":{"1min":6.931,"5min":6.931,"15min":6.5380000000000003},"max":{"1min":73.950000000000003,"5min":238.096,"15min":573.05700000000002},"last":14.992000000000001}]},{"osd":4,"last update":"Fri Dec 15 17:23:00 2023","interfaces":[{"interface":"back","average":{"1min":16.614000000000001,"5min":19.768999999999998,"15min":28.959},"min":{"1min":7.5019999999999998,"5min":7.5019999999999998,"15min":7.1159999999999997},"max":{"1min":60.703000000000003,"5min":228.708,"15min":628.97299999999996},"last":9.4979999999999993},{"interface":"front","average":{"1min":19.669,"5min":24.192,"15min":34.237000000000002},"min":{"1min":13.58,"5min":11.423,"15min":8.8239999999999998},"max":{"1min":29.742999999999999,"5min":230.202,"15min":637.40200000000004},"last":18.204999999999998}]},{"osd":5,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":14.532,"5min":20.297999999999998,"15min":26.827999999999999},"min":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.782},"max":{"1min":32.411000000000001,"5min":144.59200000000001,"15min":752.79499999999996},"last":8.6159999999999997},{"interface":"front","average":{"1min":16.890999999999998,"5min":21.039000000000001,"15min":25.788},"min":{"1min":9.5,"5min":9.2129999999999992,"15min":6.0119999999999996},"max":{"1min":45.607999999999997,"5min":145.405,"15min":263.79500000000002},"last":15.510999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":16.896000000000001,"5min":21.753,"15min":27.977},"min":{"1min":9.3599999999999994,"5min":8.0150000000000006,"15min":7.5819999999999999},"max":{"1min":40.414999999999999,"5min":134.255,"15min":498.464},"last":11.379},{"interface":"front","average":{"1min":21.16,"5min":24.161000000000001,"15min":31.126999999999999},"min":{"1min":12.519,"5min":10.362,"15min":8.6560000000000006},"max":{"1min":60.975000000000001,"5min":153.113,"15min":636.16399999999999},"last":20.210000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":17.704999999999998,"5min":23.234000000000002,"15min":31.463000000000001},"min":{"1min":11.615,"5min":10.316000000000001,"15min":8.4930000000000003},"max":{"1min":25.109000000000002,"5min":202.89500000000001,"15min":572.08699999999999},"last":16.465},{"interface":"front","average":{"1min":19.895,"5min":24.616,"15min":31.809999999999999},"min":{"1min":13.664999999999999,"5min":13.173999999999999,"15min":10.406000000000001},"max":{"1min":32.581000000000003,"5min":156.91900000000001,"15min":624.04100000000005},"last":17.247}]}]},{"osd":7,"up_from":14,"seq":60129542411,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76948,"kb_used_data":5508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294892,"statfs":{"total":96636764160,"available":96557969408,"internally_reserved":0,"allocated":5640192,"data_stored":2992117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":63,"apply_latency_ms":63,"commit_latency_ns":63000000,"apply_latency_ns":63000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":11.852,"5min":12.436999999999999,"15min":16.131},"min":{"1min":5.0750000000000002,"5min":4.3810000000000002,"15min":4.3259999999999996},"max":{"1min":37.542000000000002,"5min":91.055999999999997,"15min":161.38499999999999},"last":21.841000000000001},{"interface":"front","average":{"1min":12.800000000000001,"5min":12.664999999999999,"15min":16.463999999999999},"min":{"1min":6.5910000000000002,"5min":4.8339999999999996,"15min":4.8339999999999996},"max":{"1min":64.691999999999993,"5min":93.209999999999994,"15min":142.673},"last":7.3879999999999999}]},{"osd":1,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":20.167000000000002,"5min":18.283000000000001,"15min":22.882000000000001},"min":{"1min":7.2480000000000002,"5min":6.3550000000000004,"15min":5.7629999999999999},"max":{"1min":74.025999999999996,"5min":143.17599999999999,"15min":275.99299999999999},"last":15.551},{"interface":"front","average":{"1min":15.464,"5min":14.946,"15min":21.763999999999999},"min":{"1min":5.9749999999999996,"5min":5.7960000000000003,"15min":5.2649999999999997},"max":{"1min":81.099999999999994,"5min":88.022999999999996,"15min":342.22000000000003},"last":16.494}]},{"osd":2,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":20.966999999999999,"5min":17.347000000000001,"15min":19.687000000000001},"min":{"1min":8.2509999999999994,"5min":7.383,"15min":6.0670000000000002},"max":{"1min":86.772000000000006,"5min":95.926000000000002,"15min":131.81299999999999},"last":26.233000000000001},{"interface":"front","average":{"1min":16.437000000000001,"5min":14.654999999999999,"15min":19.41},"min":{"1min":6.9470000000000001,"5min":6.266,"15min":6.266},"max":{"1min":84.876000000000005,"5min":84.876000000000005,"15min":117.864},"last":23.754999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":20.248000000000001,"5min":20.637,"15min":24.939},"min":{"1min":6.5330000000000004,"5min":6.5330000000000004,"15min":6.4370000000000003},"max":{"1min":118.39400000000001,"5min":118.39400000000001,"15min":390.91000000000003},"last":19.044},{"interface":"front","average":{"1min":20.558,"5min":21.027999999999999,"15min":26.327999999999999},"min":{"1min":8.1479999999999997,"5min":7.4260000000000002,"15min":5.7320000000000002},"max":{"1min":138.559,"5min":138.559,"15min":244.13},"last":24.443999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:23:51 2023","interfaces":[{"interface":"back","average":{"1min":24.532,"5min":22.248999999999999,"15min":24.129999999999999},"min":{"1min":7.3529999999999998,"5min":7.3529999999999998,"15min":7.3529999999999998},"max":{"1min":144.52099999999999,"5min":144.52099999999999,"15min":245.65600000000001},"last":17.129000000000001},{"interface":"front","average":{"1min":25.827999999999999,"5min":24.245999999999999,"15min":23.055},"min":{"1min":9.4199999999999999,"5min":7.726,"15min":6.9139999999999997},"max":{"1min":143.92099999999999,"5min":189.988,"15min":193.07400000000001},"last":21.242000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":17.966000000000001,"5min":19.504000000000001,"15min":25.530000000000001},"min":{"1min":7.7699999999999996,"5min":7.7699999999999996,"15min":7.7699999999999996},"max":{"1min":69.656999999999996,"5min":109.459,"15min":198.96000000000001},"last":22.948},{"interface":"front","average":{"1min":21.565999999999999,"5min":20.701000000000001,"15min":27.978999999999999},"min":{"1min":8.7959999999999994,"5min":7.6189999999999998,"15min":7.6189999999999998},"max":{"1min":145.94,"5min":165.97999999999999,"15min":266.33699999999999},"last":17.718}]},{"osd":6,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":24.082000000000001,"5min":22.300000000000001,"15min":30.507000000000001},"min":{"1min":11.667999999999999,"5min":7.952,"15min":7.8949999999999996},"max":{"1min":112.369,"5min":113.128,"15min":380.62700000000001},"last":25.324999999999999},{"interface":"front","average":{"1min":23.890999999999998,"5min":21.359000000000002,"15min":26.186},"min":{"1min":8.3550000000000004,"5min":8.3550000000000004,"15min":7.2270000000000003},"max":{"1min":108.03,"5min":108.03,"15min":226.91499999999999},"last":18.34}]}]},{"osd":6,"up_from":14,"seq":60129542412,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76968,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294872,"statfs":{"total":96636764160,"available":96557948928,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":10.004,"5min":12.342000000000001,"15min":18.960000000000001},"min":{"1min":4.4269999999999996,"5min":4.133,"15min":4.133},"max":{"1min":25.085999999999999,"5min":128.82499999999999,"15min":223.15199999999999},"last":7.9279999999999999},{"interface":"front","average":{"1min":9.6329999999999991,"5min":14.333,"15min":18.896000000000001},"min":{"1min":6.2009999999999996,"5min":5.3490000000000002,"15min":4.6920000000000002},"max":{"1min":17.561,"5min":120.85299999999999,"15min":190.249},"last":11.704000000000001}]},{"osd":1,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":12.476000000000001,"5min":18.774000000000001,"15min":22.052},"min":{"1min":7.2279999999999998,"5min":5.1319999999999997,"15min":5.0449999999999999},"max":{"1min":24.771999999999998,"5min":177.15100000000001,"15min":292.72199999999998},"last":9.8870000000000005},{"interface":"front","average":{"1min":14.473000000000001,"5min":19.847999999999999,"15min":25.225000000000001},"min":{"1min":7.6120000000000001,"5min":6.4390000000000001,"15min":5.0309999999999997},"max":{"1min":41.270000000000003,"5min":174.02199999999999,"15min":422.36099999999999},"last":14.478}]},{"osd":2,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":17.154,"5min":17.030000000000001,"15min":19.913},"min":{"1min":7.7169999999999996,"5min":6.2850000000000001,"15min":5.2699999999999996},"max":{"1min":66.974000000000004,"5min":131.71000000000001,"15min":131.863},"last":17.501999999999999},{"interface":"front","average":{"1min":13.039,"5min":19.218,"15min":23.687000000000001},"min":{"1min":6.9530000000000003,"5min":6.9530000000000003,"15min":6.7439999999999998},"max":{"1min":65.632000000000005,"5min":112.548,"15min":189.417},"last":10.91}]},{"osd":3,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":21.212,"5min":24.562000000000001,"15min":27.657},"min":{"1min":7.7279999999999998,"5min":7.7279999999999998,"15min":7.7279999999999998},"max":{"1min":77.271000000000001,"5min":263.45299999999997,"15min":263.45299999999997},"last":16.798999999999999},{"interface":"front","average":{"1min":24.375,"5min":27.704000000000001,"15min":30.245999999999999},"min":{"1min":11.551,"5min":10.195,"15min":6.9100000000000001},"max":{"1min":83.891999999999996,"5min":264.54000000000002,"15min":356.815},"last":26.609000000000002}]},{"osd":4,"last update":"Fri Dec 15 17:23:25 2023","interfaces":[{"interface":"back","average":{"1min":17.318000000000001,"5min":20.620000000000001,"15min":24.326000000000001},"min":{"1min":6.4020000000000001,"5min":6.4020000000000001,"15min":6.4020000000000001},"max":{"1min":118.277,"5min":122.881,"15min":224.96299999999999},"last":31.928000000000001},{"interface":"front","average":{"1min":16.988,"5min":21.347000000000001,"15min":26.774000000000001},"min":{"1min":6.9100000000000001,"5min":6.9100000000000001,"15min":6.9100000000000001},"max":{"1min":99.513000000000005,"5min":127.607,"15min":266.88400000000001},"last":28.789999999999999}]},{"osd":5,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":15.765000000000001,"5min":21.884,"15min":28.344000000000001},"min":{"1min":7.9089999999999998,"5min":7.601,"15min":7.0369999999999999},"max":{"1min":40.454999999999998,"5min":164.62899999999999,"15min":349.983},"last":29.637},{"interface":"front","average":{"1min":14.981999999999999,"5min":21.09,"15min":25.739000000000001},"min":{"1min":9.1750000000000007,"5min":7.4530000000000003,"15min":6.8449999999999998},"max":{"1min":26.895,"5min":115.54900000000001,"15min":324.95299999999997},"last":12.641}]},{"osd":7,"last update":"Fri Dec 15 17:23:47 2023","interfaces":[{"interface":"back","average":{"1min":15.082000000000001,"5min":22.818999999999999,"15min":28.507000000000001},"min":{"1min":9.4640000000000004,"5min":8.2110000000000003,"15min":6.8520000000000003},"max":{"1min":38.795999999999999,"5min":137.59700000000001,"15min":268.66500000000002},"last":15.759},{"interface":"front","average":{"1min":18.492999999999999,"5min":26.584,"15min":33.052999999999997},"min":{"1min":12.459,"5min":8.8849999999999998,"15min":8.8849999999999998},"max":{"1min":48.68,"5min":135.74199999999999,"15min":318.56900000000002},"last":18.266999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542413,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81528,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290312,"statfs":{"total":96636764160,"available":96553279488,"internally_reserved":0,"allocated":6066176,"data_stored":3403689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":10.813000000000001,"5min":18.579999999999998,"15min":19.327999999999999},"min":{"1min":5.0339999999999998,"5min":4.7380000000000004,"15min":4.2859999999999996},"max":{"1min":24.373000000000001,"5min":209.15799999999999,"15min":223.69499999999999},"last":5.2530000000000001},{"interface":"front","average":{"1min":10.352,"5min":17.178999999999998,"15min":19.341999999999999},"min":{"1min":4.2590000000000003,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":17.353000000000002,"5min":183.93899999999999,"15min":226.76300000000001},"last":8.766}]},{"osd":1,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":13.563000000000001,"5min":17.940000000000001,"15min":22.196000000000002},"min":{"1min":6.6539999999999999,"5min":5.1260000000000003,"15min":4.7839999999999998},"max":{"1min":63.030999999999999,"5min":137.07499999999999,"15min":340.32999999999998},"last":7.8949999999999996},{"interface":"front","average":{"1min":18.015999999999998,"5min":20.047999999999998,"15min":24.577000000000002},"min":{"1min":8.7149999999999999,"5min":5.7850000000000001,"15min":5.6550000000000002},"max":{"1min":57.795000000000002,"5min":169.648,"15min":326.375},"last":11.550000000000001}]},{"osd":2,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":17.568000000000001,"5min":20.483000000000001,"15min":22.577000000000002},"min":{"1min":6.6630000000000003,"5min":6.6630000000000003,"15min":5.3979999999999997},"max":{"1min":92.230000000000004,"5min":174.09100000000001,"15min":240.00999999999999},"last":15.416},{"interface":"front","average":{"1min":21.774999999999999,"5min":24.068000000000001,"15min":25.559000000000001},"min":{"1min":6.7690000000000001,"5min":5.569,"15min":5.569},"max":{"1min":128.98099999999999,"5min":228.31899999999999,"15min":371.30099999999999},"last":10.891}]},{"osd":3,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":25.259,"5min":25.724,"15min":28.169},"min":{"1min":9.2319999999999993,"5min":6.5999999999999996,"15min":6.5999999999999996},"max":{"1min":100.711,"5min":177.51300000000001,"15min":321.26100000000002},"last":10.199},{"interface":"front","average":{"1min":25.388999999999999,"5min":26.262,"15min":27.506},"min":{"1min":7.3799999999999999,"5min":7.101,"15min":6.8730000000000002},"max":{"1min":101.599,"5min":172.887,"15min":284.12599999999998},"last":12.177}]},{"osd":4,"last update":"Fri Dec 15 17:23:40 2023","interfaces":[{"interface":"back","average":{"1min":18.481000000000002,"5min":29.667999999999999,"15min":27.547000000000001},"min":{"1min":8.3559999999999999,"5min":8.3559999999999999,"15min":6.8600000000000003},"max":{"1min":61.365000000000002,"5min":229.077,"15min":376.39800000000002},"last":13.378},{"interface":"front","average":{"1min":19.219000000000001,"5min":28.552,"15min":27.09},"min":{"1min":10.989000000000001,"5min":7.8840000000000003,"15min":6.2789999999999999},"max":{"1min":55.509999999999998,"5min":211.03100000000001,"15min":338.56400000000002},"last":16.888999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":25.266999999999999,"15min":27.039000000000001},"min":{"1min":12.606999999999999,"5min":7.8079999999999998,"15min":7.0789999999999997},"max":{"1min":57.170000000000002,"5min":215.21299999999999,"15min":378.327},"last":16.372},{"interface":"front","average":{"1min":21.053000000000001,"5min":29.216999999999999,"15min":28.443000000000001},"min":{"1min":9.3049999999999997,"5min":8.8900000000000006,"15min":6.6369999999999996},"max":{"1min":62.055999999999997,"5min":228.58799999999999,"15min":407.16500000000002},"last":14.061}]},{"osd":7,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":19.960999999999999,"5min":24.295000000000002,"15min":28.638999999999999},"min":{"1min":12.951000000000001,"5min":10.864000000000001,"15min":7.7599999999999998},"max":{"1min":39.515999999999998,"5min":151.386,"15min":292.82299999999998},"last":17.649000000000001},{"interface":"front","average":{"1min":20.140000000000001,"5min":27.221,"15min":30.507000000000001},"min":{"1min":11.416,"5min":9.2170000000000005,"15min":7.8650000000000002},"max":{"1min":28.832999999999998,"5min":190.74600000000001,"15min":312.73599999999999},"last":18.655000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542413,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58904,"kb_used_data":5508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312936,"statfs":{"total":96636764160,"available":96576446464,"internally_reserved":0,"allocated":5640192,"data_stored":2992117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":20.422000000000001,"5min":13.071,"15min":15.058},"min":{"1min":4.7779999999999996,"5min":4.4560000000000004,"15min":4.1289999999999996},"max":{"1min":62.142000000000003,"5min":62.142000000000003,"15min":193.11500000000001},"last":10.169},{"interface":"front","average":{"1min":27.135999999999999,"5min":13.522,"15min":16.253},"min":{"1min":5.9699999999999998,"5min":4.4109999999999996,"15min":4.3719999999999999},"max":{"1min":96.224000000000004,"5min":96.224000000000004,"15min":235.09800000000001},"last":6.1749999999999998}]},{"osd":1,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":26.41,"5min":16.433,"15min":19.759},"min":{"1min":8.8640000000000008,"5min":5.2199999999999998,"15min":4.9930000000000003},"max":{"1min":86.936999999999998,"5min":86.936999999999998,"15min":289.11099999999999},"last":14.173999999999999},{"interface":"front","average":{"1min":22.891999999999999,"5min":18.584,"15min":21.152999999999999},"min":{"1min":7.3520000000000003,"5min":6.2549999999999999,"15min":6.2249999999999996},"max":{"1min":83.016999999999996,"5min":130.535,"15min":218.32400000000001},"last":9.3599999999999994}]},{"osd":2,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":35.674999999999997,"5min":20.347999999999999,"15min":18.050000000000001},"min":{"1min":8.2569999999999997,"5min":6.8179999999999996,"15min":5.5880000000000001},"max":{"1min":163.31399999999999,"5min":163.31399999999999,"15min":163.31399999999999},"last":25.213999999999999},{"interface":"front","average":{"1min":31.010000000000002,"5min":19.277999999999999,"15min":18.506},"min":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":5.4690000000000003},"max":{"1min":134.78700000000001,"5min":134.78700000000001,"15min":140.416},"last":11.577}]},{"osd":3,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":26.495000000000001,"5min":23.353000000000002,"15min":21.949000000000002},"min":{"1min":11.071999999999999,"5min":8.9169999999999998,"15min":7.2450000000000001},"max":{"1min":89.188000000000002,"5min":230.505,"15min":230.505},"last":24.515000000000001},{"interface":"front","average":{"1min":22.827000000000002,"5min":19.329999999999998,"15min":21.117999999999999},"min":{"1min":9.0299999999999994,"5min":6.4210000000000003,"15min":6.4210000000000003},"max":{"1min":84.497,"5min":136.66499999999999,"15min":203.99100000000001},"last":12.701000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":19.713000000000001,"5min":19.503,"15min":21.353999999999999},"min":{"1min":7.7530000000000001,"5min":6.9930000000000003,"15min":6.9930000000000003},"max":{"1min":87.989999999999995,"5min":123.71599999999999,"15min":162.02099999999999},"last":13.314},{"interface":"front","average":{"1min":23.210999999999999,"5min":23.539999999999999,"15min":24.937999999999999},"min":{"1min":10.042999999999999,"5min":10.042999999999999,"15min":7.1130000000000004},"max":{"1min":89.099999999999994,"5min":159.16999999999999,"15min":162.922},"last":28.855}]},{"osd":6,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":25.030000000000001,"5min":19.335999999999999,"15min":22.289999999999999},"min":{"1min":7.5579999999999998,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":84.040000000000006,"5min":85.606999999999999,"15min":744.33199999999999},"last":30.443999999999999},{"interface":"front","average":{"1min":38.479999999999997,"5min":26.033000000000001,"15min":25.606000000000002},"min":{"1min":14.532999999999999,"5min":10.544,"15min":9.1799999999999997},"max":{"1min":157.441,"5min":157.441,"15min":251.19200000000001},"last":29.472000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":31.425999999999998,"5min":25.463000000000001,"15min":26.760999999999999},"min":{"1min":17.059000000000001,"5min":11.148999999999999,"15min":9.7330000000000005},"max":{"1min":91.006,"5min":144.72,"15min":202.46600000000001},"last":27.286999999999999},{"interface":"front","average":{"1min":21.945,"5min":18.006,"15min":20.908000000000001},"min":{"1min":9.6690000000000005,"5min":7.5860000000000003,"15min":6.5430000000000001},"max":{"1min":78.950000000000003,"5min":85.629999999999995,"15min":199.01400000000001},"last":10.967000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-15T17:24:00.555 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.652+0000 7f9e014de640 10 monclient: get_monmap_and_config 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.652+0000 7f9e014de640 10 monclient: build_initial_monmap 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.652+0000 7f9e014de640 10 monclient: monmap: 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:00.654118+0000 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:00.654118+0000 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:00.654 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:00.655 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:00.655 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 1 Processor -- start 2023-12-15T17:24:00.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 1 -- start start 2023-12-15T17:24:00.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 10 monclient: init 2023-12-15T17:24:00.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:00.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-12-15T17:24:00.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9dfc0a8910 addr v1:172.21.15.69:6789/0 2023-12-15T17:24:00.656 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9dfc0a9b70 0x7f9dfc13f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 10 monclient(hunting): picked mon.noname-a con 0x7f9dfc0a9b70 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9dfc141f90 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc0a8cf0 con 0x7f9dfc0a8910 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc0aabd0 con 0x7f9dfc141f90 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dfc0aaeb0 con 0x7f9dfc0a9b70 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.653+0000 7f9e014de640 10 monclient(hunting): authenticate will time out at 2744.970255s 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.654+0000 7f9df27fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9dfc0a9b70 0x7f9dfc13f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.654+0000 7f9dfb7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f9dfc141f90 0x7f9dfc1423f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:45422/0 (socket says 172.21.15.69:45422) 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.654+0000 7f9dfb7fe640 1 -- 172.21.15.69:0/1840482148 learned_addr learned my addr 172.21.15.69:0/1840482148 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.654+0000 7f9df27fc640 10 monclient(hunting): get_auth_request con 0x7f9dfc0a9b70 auth_method 0 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.654+0000 7f9df27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.654+0000 7f9df27fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.654+0000 7f9df27fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.655+0000 7f9df27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.655+0000 7f9df27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.655+0000 7f9df27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9dfa7fc640 1 -- 172.21.15.69:0/1840482148 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070432505 0 0) 0x7f9dfc0aabd0 con 0x7f9dfc141f90 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9dfa7fc640 10 monclient(hunting): _negotiate 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9dfa7fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9dfa7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9dfa7fc640 10 monclient(hunting): my global_id is 21424 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9dfa7fc640 1 -- 172.21.15.69:0/1840482148 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de8003520 con 0x7f9dfc141f90 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9df27fc640 10 monclient(hunting): handle_auth_done global_id 16062 payload 274 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9df27fc640 1 -- 172.21.15.69:0/1840482148 >> v1:172.21.15.192:6789/0 conn(0x7f9dfc141f90 legacy=0x7f9dfc1423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9df27fc640 1 -- 172.21.15.69:0/1840482148 >> v1:172.21.15.69:6789/0 conn(0x7f9dfc0a8910 legacy=0x7f9dfc13ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9df27fc640 10 monclient: _finish_hunting 0 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9df27fc640 1 monclient: found mon.noname-a 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9df27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9df27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:24:00.659 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.657+0000 7f9df27fc640 1 -- 172.21.15.69:0/1840482148 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc0a9720 con 0x7f9dfc0a9b70 2023-12-15T17:24:00.660 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.659+0000 7f9df27fc640 1 --2- 172.21.15.69:0/1840482148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc0a9b70 0x7f9dfc13f370 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7f9de40097a0 tx=0x7f9de402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583bc44f16bc8cf0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:00.664 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 1 -- 172.21.15.69:0/1840482148 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4031830 con 0x7f9dfc0a9b70 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 10 monclient: dump: 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 10 monclient: _finish_auth 0 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 10 monclient: _check_auth_tickets 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 1 -- 172.21.15.69:0/1840482148 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de4031990 con 0x7f9dfc0a9b70 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9e014de640 5 monclient: authenticate success, global_id 16062 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9e014de640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9e014de640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9e014de640 4 set_mon_vals no callback set 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 1 -- 172.21.15.69:0/1840482148 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4031d20 con 0x7f9dfc0a9b70 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.663+0000 7f9dfa7fc640 10 monclient: dump: 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:00.667 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.664+0000 7f9e014de640 10 monclient: shutdown 2023-12-15T17:24:00.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.664+0000 7f9e014de640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:00.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.664+0000 7f9e014de640 1 -- 172.21.15.69:0/1840482148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc0a9b70 msgr2=0x7f9dfc13f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:00.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.664+0000 7f9e014de640 1 --2- 172.21.15.69:0/1840482148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc0a9b70 0x7f9dfc13f370 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7f9de40097a0 tx=0x7f9de402b8d0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.664+0000 7f9e014de640 1 -- 172.21.15.69:0/1840482148 shutdown_connections 2023-12-15T17:24:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.664+0000 7f9e014de640 1 --2- 172.21.15.69:0/1840482148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc0a9b70 0x7f9dfc13f370 unknown :-1 s=CLOSED pgs=3158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.664+0000 7f9e014de640 1 -- 172.21.15.69:0/1840482148 >> 172.21.15.69:0/1840482148 conn(0x7f9dfc0b7780 msgr2=0x7f9dfc0b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.664+0000 7f9e014de640 1 -- 172.21.15.69:0/1840482148 shutdown_connections 2023-12-15T17:24:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.664+0000 7f9e014de640 1 -- 172.21.15.69:0/1840482148 wait complete. 2023-12-15T17:24:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.664+0000 7f9e014de640 10 monclient: build_initial_monmap 2023-12-15T17:24:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.664+0000 7f9e014de640 10 monclient: monmap: 2023-12-15T17:24:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 librados: starting msgr at 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 librados: starting objecter 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 Processor -- start 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 -- start start 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 librados: setting wanted keys 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 librados: calling monclient init 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 10 monclient: init 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:24:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dfc0a8910 0x7f9dfc13a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 10 monclient(hunting): picked mon.noname-b con 0x7f9dfc0a8910 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dfc141f90 0x7f9dfc0ac3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 10 monclient(hunting): picked mon.noname-c con 0x7f9dfc141f90 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc13b030 0x7f9dfc13b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 10 monclient(hunting): picked mon.noname-a con 0x7f9dfc13b030 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc13a880 con 0x7f9dfc13b030 2023-12-15T17:24:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc0ac910 con 0x7f9dfc0a8910 2023-12-15T17:24:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc13ba00 con 0x7f9dfc141f90 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.665+0000 7f9e014de640 10 monclient(hunting): authenticate will time out at 2744.982329s 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9df27fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dfc141f90 0x7f9dfc0ac3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfb7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc13b030 0x7f9dfc13b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfaffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dfc0a8910 0x7f9dfc13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfb7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc13b030 0x7f9dfc13b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45326/0 (socket says 172.21.15.69:45326) 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfb7fe640 1 -- 172.21.15.69:0/1932982028 learned_addr learned my addr 172.21.15.69:0/1932982028 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f9dfc13b030 auth_method 0 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfb7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfaffd640 10 monclient(hunting): get_auth_request con 0x7f9dfc0a8910 auth_method 0 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfaffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9dfaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9df27fc640 10 monclient(hunting): get_auth_request con 0x7f9dfc141f90 auth_method 0 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9df27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9df27fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.666+0000 7f9df27fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.667+0000 7f9dfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.667+0000 7f9dfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.667+0000 7f9dfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.667+0000 7f9dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.667+0000 7f9dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.667+0000 7f9dfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.667+0000 7f9df27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.667+0000 7f9df27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.667+0000 7f9df27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:00.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.670+0000 7f9dfaffd640 10 monclient(hunting): handle_auth_done global_id 21427 payload 824 2023-12-15T17:24:00.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.670+0000 7f9dfaffd640 1 -- 172.21.15.69:0/1932982028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dfc141f90 msgr2=0x7f9dfc0ac3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:00.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.670+0000 7f9dfaffd640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dfc141f90 0x7f9dfc0ac3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.670+0000 7f9dfaffd640 1 -- 172.21.15.69:0/1932982028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc13b030 msgr2=0x7f9dfc13b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:00.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.670+0000 7f9dfaffd640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc13b030 0x7f9dfc13b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.670+0000 7f9dfaffd640 10 monclient: _finish_hunting 0 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.670+0000 7f9dfaffd640 1 monclient: found mon.noname-b 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.670+0000 7f9dfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.670+0000 7f9dfaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.670+0000 7f9dfaffd640 1 -- 172.21.15.69:0/1932982028 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de4009400 con 0x7f9dfc0a8910 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.672+0000 7f9dfaffd640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dfc0a8910 0x7f9dfc13a340 secure :-1 s=READY pgs=5102 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00b3e0 tx=0x7f9dec00b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.672+0000 7f9df3fff640 10 client.?.objecter ms_handle_connect 0x7f9dfc0a8910 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.672+0000 7f9df3fff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9df3fff640 1 -- 172.21.15.69:0/1932982028 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec011830 con 0x7f9dfc0a8910 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9df3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9df3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9df3fff640 10 monclient: dump: 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:00.678 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9df3fff640 10 monclient: _finish_auth 0 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9df3fff640 10 monclient: _check_auth_tickets 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9df3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9df3fff640 1 -- 172.21.15.69:0/1932982028 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dec011990 con 0x7f9dfc0a8910 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9df3fff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9df3fff640 1 -- 172.21.15.69:0/1932982028 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec011d40 con 0x7f9dfc0a8910 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9e014de640 5 monclient: authenticate success, global_id 21427 2023-12-15T17:24:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9e014de640 10 monclient: _renew_subs 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9e014de640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9e014de640 1 -- 172.21.15.69:0/1932982028 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dfc0abfb0 con 0x7f9dfc0a8910 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.675+0000 7f9df9ffb640 4 set_mon_vals no callback set 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9df3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9df3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9df3fff640 10 monclient: dump: 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:00.680 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:00.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9e014de640 10 client.21427.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:00.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9e014de640 10 monclient: _renew_subs 2023-12-15T17:24:00.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9e014de640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:00.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9e014de640 1 -- 172.21.15.69:0/1932982028 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dfc1390c0 con 0x7f9dfc0a8910 2023-12-15T17:24:00.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9e014de640 1 librados: init done 2023-12-15T17:24:00.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9e014de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:00.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9e014de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:00.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9e014de640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:00.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.676+0000 7f9e014de640 1 -- 172.21.15.69:0/1932982028 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dfc13ac40 con 0x7f9dfc0a8910 2023-12-15T17:24:00.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.694+0000 7f9df3fff640 1 -- 172.21.15.69:0/1932982028 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9dec01a400 con 0x7f9dfc0a8910 2023-12-15T17:24:00.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.695+0000 7f9df3fff640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ddc06cba0 0x7f9ddc06f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.695+0000 7f9df27fc640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ddc06cba0 0x7f9ddc06f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.695+0000 7f9df27fc640 10 monclient: get_auth_request con 0x7f9ddc06cba0 auth_method 0 2023-12-15T17:24:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.695+0000 7f9df27fc640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ddc06cba0 0x7f9ddc06f060 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9de4036000 tx=0x7f9de403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.695+0000 7f9df3fff640 10 client.21427.objecter ms_handle_connect 0x7f9ddc06cba0 2023-12-15T17:24:00.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.697+0000 7f9df3fff640 1 -- 172.21.15.69:0/1932982028 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 534..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7f9dec058a90 con 0x7f9dfc0a8910 2023-12-15T17:24:00.698 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.697+0000 7f9df3fff640 10 client.21427.objecter ms_dispatch 0x7f9dfc000b90 osd_map(908..908 src has 534..908) v4 2023-12-15T17:24:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.697+0000 7f9df3fff640 3 client.21427.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.697+0000 7f9df3fff640 3 client.21427.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.697+0000 7f9df3fff640 20 client.21427.objecter dump_active .. 0 homeless 2023-12-15T17:24:00.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.931+0000 7f9df3fff640 1 -- 172.21.15.69:0/1932982028 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9dec054dd0 con 0x7f9dfc0a8910 2023-12-15T17:24:00.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.931+0000 7f9df3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:00.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:00.931+0000 7f9df3fff640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:01.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.097+0000 7f9e014de640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:24:01.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.097+0000 7f9e014de640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:24:01.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.097+0000 7f9e014de640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:01.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.097+0000 7f9e014de640 1 -- 172.21.15.69:0/1932982028 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9dfc075b40 con 0x7f9dfc0a8910 2023-12-15T17:24:01.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.133+0000 7f9df3fff640 1 -- 172.21.15.69:0/1932982028 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v908) v1 ==== 74+0+17359 (secure 0 0 0) 0x7f9dec015020 con 0x7f9dfc0a8910 2023-12-15T17:24:01.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.133+0000 7f9df3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-15T17:24:01.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.133+0000 7f9df3fff640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:24:01.134 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:24:01.134 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":908,"fsid":"1a9f4b8c-183d-4435-93ef-5057f2b4aded","created":"2023-12-15T17:01:02.006221+0000","modified":"2023-12-15T17:23:58.527884+0000","last_up_change":"2023-12-15T17:01:50.252516+0000","last_in_change":"2023-12-15T17:01:08.340674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":97,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-15T17:01:49.711473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-15T17:01:57.539457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-12-15T17:03:21.481891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51cce62a-d177-4e4b-a814-7e537407f397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":901,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6817","nonce":541068705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6819","nonce":541068705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6823","nonce":541068705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":541068705},{"type":"v1","addr":"172.21.15.69:6821","nonce":541068705}]},"public_addr":"172.21.15.69:6817/541068705","cluster_addr":"172.21.15.69:6819/541068705","heartbeat_back_addr":"172.21.15.69:6823/541068705","heartbeat_front_addr":"172.21.15.69:6821/541068705","state":["exists","up"]},{"osd":1,"uuid":"1556db3a-b386-444f-8327-3dce2ad46d33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":901,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6809","nonce":1683202761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6811","nonce":1683202761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6815","nonce":1683202761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1683202761},{"type":"v1","addr":"172.21.15.69:6813","nonce":1683202761}]},"public_addr":"172.21.15.69:6809/1683202761","cluster_addr":"172.21.15.69:6811/1683202761","heartbeat_back_addr":"172.21.15.69:6815/1683202761","heartbeat_front_addr":"172.21.15.69:6813/1683202761","state":["exists","up"]},{"osd":2,"uuid":"eaac40b5-9fc0-4e13-a322-763a9f3b685f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":901,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6801","nonce":3322029776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6803","nonce":3322029776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6807","nonce":3322029776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3322029776},{"type":"v1","addr":"172.21.15.69:6805","nonce":3322029776}]},"public_addr":"172.21.15.69:6801/3322029776","cluster_addr":"172.21.15.69:6803/3322029776","heartbeat_back_addr":"172.21.15.69:6807/3322029776","heartbeat_front_addr":"172.21.15.69:6805/3322029776","state":["exists","up"]},{"osd":3,"uuid":"5bb7974a-6c1d-4c57-9ed6-b79a70ec84ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":901,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6825","nonce":882900488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6827","nonce":882900488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6831","nonce":882900488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":882900488},{"type":"v1","addr":"172.21.15.69:6829","nonce":882900488}]},"public_addr":"172.21.15.69:6825/882900488","cluster_addr":"172.21.15.69:6827/882900488","heartbeat_back_addr":"172.21.15.69:6831/882900488","heartbeat_front_addr":"172.21.15.69:6829/882900488","state":["exists","up"]},{"osd":4,"uuid":"71221e7e-8713-4fc4-bef1-c1cbe40627da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":897,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6803","nonce":1420550374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6805","nonce":1420550374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6809","nonce":1420550374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1420550374},{"type":"v1","addr":"172.21.15.192:6807","nonce":1420550374}]},"public_addr":"172.21.15.192:6803/1420550374","cluster_addr":"172.21.15.192:6805/1420550374","heartbeat_back_addr":"172.21.15.192:6809/1420550374","heartbeat_front_addr":"172.21.15.192:6807/1420550374","state":["exists","up"]},{"osd":5,"uuid":"2c9feecc-d3b5-429d-98fe-e6eddc91eb8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":901,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6811","nonce":2003653981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6813","nonce":2003653981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6817","nonce":2003653981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2003653981},{"type":"v1","addr":"172.21.15.192:6815","nonce":2003653981}]},"public_addr":"172.21.15.192:6811/2003653981","cluster_addr":"172.21.15.192:6813/2003653981","heartbeat_back_addr":"172.21.15.192:6817/2003653981","heartbeat_front_addr":"172.21.15.192:6815/2003653981","state":["exists","up"]},{"osd":6,"uuid":"5c31d7a0-e3dc-4eb3-a554-949b1b1a8fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":901,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6819","nonce":2563835645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6821","nonce":2563835645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6825","nonce":2563835645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2563835645},{"type":"v1","addr":"172.21.15.192:6823","nonce":2563835645}]},"public_addr":"172.21.15.192:6819/2563835645","cluster_addr":"172.21.15.192:6821/2563835645","heartbeat_back_addr":"172.21.15.192:6825/2563835645","heartbeat_front_addr":"172.21.15.192:6823/2563835645","state":["exists","up"]},{"osd":7,"uuid":"ac6e8a35-6df7-48d4-b59a-1f87eceaacfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":901,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6827","nonce":4283271495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6829","nonce":4283271495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6833","nonce":4283271495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":4283271495},{"type":"v1","addr":"172.21.15.192:6831","nonce":4283271495}]},"public_addr":"172.21.15.192:6827/4283271495","cluster_addr":"172.21.15.192:6829/4283271495","heartbeat_back_addr":"172.21.15.192:6833/4283271495","heartbeat_front_addr":"172.21.15.192:6831/4283271495","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.990860+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.734395+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:47.257292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:47.560872+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.054415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.592183+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.562550+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-15T17:01:46.670431+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":9,"length":2}]},{"pool":17,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":46,"snaps":[{"begin":11,"length":2}]},{"pool":71,"snaps":[{"begin":7,"length":1}]},{"pool":92,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":95,"snaps":[{"begin":2,"length":1}]},{"pool":111,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":121,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":171,"snaps":[{"begin":12,"length":1}]},{"pool":183,"snaps":[{"begin":14,"length":1}]},{"pool":188,"snaps":[{"begin":17,"length":1}]},{"pool":191,"snaps":[{"begin":19,"length":2}]},{"pool":197,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T17:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 10 librados: watch_flush enter 2023-12-15T17:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 10 librados: watch_flush exit 2023-12-15T17:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 20 client.21427.objecter shutdown clearing up homeless session... 2023-12-15T17:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 10 client.21427.objecter successfully canceled tick 2023-12-15T17:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 -- 172.21.15.69:0/1932982028 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ddc06cba0 msgr2=0x7f9ddc06f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ddc06cba0 0x7f9ddc06f060 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9de4036000 tx=0x7f9de403c040 comp rx=0 tx=0).stop 2023-12-15T17:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 10 monclient: shutdown 2023-12-15T17:24:01.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 -- 172.21.15.69:0/1932982028 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dfc0a8910 msgr2=0x7f9dfc13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dfc0a8910 0x7f9dfc13a340 secure :-1 s=READY pgs=5102 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00b3e0 tx=0x7f9dec00b8a0 comp rx=0 tx=0).stop 2023-12-15T17:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 -- 172.21.15.69:0/1932982028 shutdown_connections 2023-12-15T17:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f9ddc06cba0 0x7f9ddc06f060 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc13b030 0x7f9dfc13b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9dfc141f90 0x7f9dfc0ac3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 --2- 172.21.15.69:0/1932982028 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dfc0a8910 0x7f9dfc13a340 unknown :-1 s=CLOSED pgs=5102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 -- 172.21.15.69:0/1932982028 >> 172.21.15.69:0/1932982028 conn(0x7f9dfc0b7780 msgr2=0x7f9dfc0b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 -- 172.21.15.69:0/1932982028 shutdown_connections 2023-12-15T17:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 -- 172.21.15.69:0/1932982028 wait complete. 2023-12-15T17:24:01.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:01.134+0000 7f9e014de640 1 librados: shutdown 2023-12-15T17:24:02.148 INFO:tasks.ceph:Scrubbing osd.0 2023-12-15T17:24:02.149 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-15T17:24:02.253 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.251+0000 7f50ebfff640 10 monclient: get_monmap_and_config 2023-12-15T17:24:02.253 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.251+0000 7f50ebfff640 10 monclient: build_initial_monmap 2023-12-15T17:24:02.253 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.251+0000 7f50ebfff640 10 monclient: monmap: 2023-12-15T17:24:02.253 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:02.253 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:02.253 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:02.253279+0000 2023-12-15T17:24:02.253 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:02.253279+0000 2023-12-15T17:24:02.253 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:02.253 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.254 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:02.254 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:02.254 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:02.254 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:02.254 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:02.254 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.252+0000 7f50ebfff640 1 Processor -- start 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.252+0000 7f50ebfff640 1 -- start start 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.252+0000 7f50ebfff640 10 monclient: init 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 0x7f50ec142280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 10 monclient(hunting): picked mon.noname-b con 0x7f50ec0a8710 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50ec1427c0 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f50ec0a8af0 0x7f50ec13f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 10 monclient(hunting): picked mon.noname-a con 0x7f50ec0a8af0 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec0aabe0 con 0x7f50ec1427c0 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:02.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ec0aa9e0 con 0x7f50ec0a8af0 2023-12-15T17:24:02.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:02.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec141d60 con 0x7f50ec0a8710 2023-12-15T17:24:02.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:02.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50ebfff640 10 monclient(hunting): authenticate will time out at 2746.569485s 2023-12-15T17:24:02.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50eb7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f50ec0a8af0 0x7f50ec13f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.253+0000 7f50eaffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 0x7f50ec142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eaffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 0x7f50ec142280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42288/0 (socket says 172.21.15.69:42288) 2023-12-15T17:24:02.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eb7fe640 10 monclient(hunting): get_auth_request con 0x7f50ec0a8af0 auth_method 0 2023-12-15T17:24:02.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eaffd640 1 -- 172.21.15.69:0/2579367968 learned_addr learned my addr 172.21.15.69:0/2579367968 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eb7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eaffd640 10 monclient(hunting): get_auth_request con 0x7f50ec0a8710 auth_method 0 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eaffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:02.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:02.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.254+0000 7f50eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:02.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.256+0000 7f50eb7fe640 10 monclient(hunting): handle_auth_done global_id 16068 payload 274 2023-12-15T17:24:02.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.256+0000 7f50eb7fe640 1 -- 172.21.15.69:0/2579367968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 msgr2=0x7f50ec142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:02.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.256+0000 7f50eb7fe640 1 --2- 172.21.15.69:0/2579367968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 0x7f50ec142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.256+0000 7f50eb7fe640 1 -- 172.21.15.69:0/2579367968 >> v1:172.21.15.192:6789/0 conn(0x7f50ec1427c0 legacy=0x7f50ec13ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.256+0000 7f50eb7fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:02.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.256+0000 7f50eb7fe640 1 monclient: found mon.noname-a 2023-12-15T17:24:02.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.256+0000 7f50eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:02.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.256+0000 7f50eb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:24:02.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.256+0000 7f50eb7fe640 1 -- 172.21.15.69:0/2579367968 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0a9930 con 0x7f50ec0a8af0 2023-12-15T17:24:02.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.256+0000 7f50eaffd640 1 --2- 172.21.15.69:0/2579367968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 0x7f50ec142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-15T17:24:02.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.258+0000 7f50eb7fe640 1 --2- 172.21.15.69:0/2579367968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0a8af0 0x7f50ec13f480 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009d90 tx=0x7f50e402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed92fc3663909115 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:02.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 1 -- 172.21.15.69:0/2579367968 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e402cc50 con 0x7f50ec0a8af0 2023-12-15T17:24:02.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:02.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:24:02.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 10 monclient: dump: 2023-12-15T17:24:02.263 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:02.263 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:02.263 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.263 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 10 monclient: _finish_auth 0 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 10 monclient: _check_auth_tickets 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 1 -- 172.21.15.69:0/2579367968 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e402cdb0 con 0x7f50ec0a8af0 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50ebfff640 5 monclient: authenticate success, global_id 16068 2023-12-15T17:24:02.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50ebfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50ebfff640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 1 -- 172.21.15.69:0/2579367968 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e4034660 con 0x7f50ec0a8af0 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50ebfff640 4 set_mon_vals no callback set 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.261+0000 7f50e9ffb640 10 monclient: dump: 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.266 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:02.266 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:02.266 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:02.266 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.262+0000 7f50ebfff640 10 monclient: shutdown 2023-12-15T17:24:02.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.262+0000 7f50ebfff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:02.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.262+0000 7f50ebfff640 1 -- 172.21.15.69:0/2579367968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0a8af0 msgr2=0x7f50ec13f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.262+0000 7f50ebfff640 1 --2- 172.21.15.69:0/2579367968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0a8af0 0x7f50ec13f480 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009d90 tx=0x7f50e402bea0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.262+0000 7f50ebfff640 1 -- 172.21.15.69:0/2579367968 shutdown_connections 2023-12-15T17:24:02.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.262+0000 7f50ebfff640 1 --2- 172.21.15.69:0/2579367968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0a8af0 0x7f50ec13f480 unknown :-1 s=CLOSED pgs=3159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.262+0000 7f50ebfff640 1 --2- 172.21.15.69:0/2579367968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 0x7f50ec142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.262+0000 7f50ebfff640 1 -- 172.21.15.69:0/2579367968 >> 172.21.15.69:0/2579367968 conn(0x7f50ec0617a0 msgr2=0x7f50ec0b7600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.262+0000 7f50ebfff640 1 -- 172.21.15.69:0/2579367968 shutdown_connections 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.262+0000 7f50ebfff640 1 -- 172.21.15.69:0/2579367968 wait complete. 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.263+0000 7f50ebfff640 10 monclient: build_initial_monmap 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.263+0000 7f50ebfff640 10 monclient: monmap: 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.263+0000 7f50ebfff640 1 librados: starting msgr at 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.263+0000 7f50ebfff640 1 librados: starting objecter 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.263+0000 7f50ebfff640 1 Processor -- start 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 1 -- start start 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 1 librados: setting wanted keys 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 1 librados: calling monclient init 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 10 monclient: init 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 0x7f50ec145b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 10 monclient(hunting): picked mon.noname-c con 0x7f50ec0a8710 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0a8af0 0x7f50ec137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 10 monclient(hunting): picked mon.noname-a con 0x7f50ec0a8af0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50ec1427c0 0x7f50ec1da7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 10 monclient(hunting): picked mon.noname-b con 0x7f50ec1427c0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec1da310 con 0x7f50ec0a8af0 2023-12-15T17:24:02.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec144900 con 0x7f50ec1427c0 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec142ec0 con 0x7f50ec0a8710 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.264+0000 7f50ebfff640 10 monclient(hunting): authenticate will time out at 2746.580971s 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eaffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 0x7f50ec145b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50ec1427c0 0x7f50ec1da7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eaffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 0x7f50ec145b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42302/0 (socket says 172.21.15.69:42302) 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50ec1427c0 0x7f50ec1da7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:43866/0 (socket says 172.21.15.69:43866) 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eaffd640 1 -- 172.21.15.69:0/495963294 learned_addr learned my addr 172.21.15.69:0/495963294 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eaffd640 10 monclient(hunting): get_auth_request con 0x7f50ec0a8710 auth_method 0 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eaffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eb7fe640 10 monclient(hunting): get_auth_request con 0x7f50ec1427c0 auth_method 0 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eb7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.265+0000 7f50eb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.266+0000 7f50eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.266+0000 7f50eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.266+0000 7f50eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.266+0000 7f50eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.266+0000 7f50eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.266+0000 7f50eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.266+0000 7f50ea7fc640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0a8af0 0x7f50ec137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.267+0000 7f50ea7fc640 10 monclient(hunting): get_auth_request con 0x7f50ec0a8af0 auth_method 0 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.267+0000 7f50ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.267+0000 7f50ea7fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.267+0000 7f50ea7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:02.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.267+0000 7f50ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:02.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.267+0000 7f50ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:02.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.267+0000 7f50ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.268+0000 7f50eb7fe640 10 monclient(hunting): handle_auth_done global_id 21433 payload 824 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.268+0000 7f50eb7fe640 1 -- 172.21.15.69:0/495963294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 msgr2=0x7f50ec145b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.268+0000 7f50eb7fe640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec0a8710 0x7f50ec145b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.268+0000 7f50eb7fe640 1 -- 172.21.15.69:0/495963294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0a8af0 msgr2=0x7f50ec137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.268+0000 7f50eb7fe640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0a8af0 0x7f50ec137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.268+0000 7f50eb7fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.268+0000 7f50eb7fe640 1 monclient: found mon.noname-b 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.268+0000 7f50eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.268+0000 7f50eb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.268+0000 7f50eb7fe640 1 -- 172.21.15.69:0/495963294 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc0094d0 con 0x7f50ec1427c0 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.271+0000 7f50eb7fe640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50ec1427c0 0x7f50ec1da7d0 secure :-1 s=READY pgs=5104 cs=0 l=1 rev1=1 crypto rx=0x7f50e4033000 tx=0x7f50e4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:02.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.271+0000 7f50d37fe640 10 client.?.objecter ms_handle_connect 0x7f50ec1427c0 2023-12-15T17:24:02.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.271+0000 7f50d37fe640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:02.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.274+0000 7f50d37fe640 1 -- 172.21.15.69:0/495963294 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e403e610 con 0x7f50ec1427c0 2023-12-15T17:24:02.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.274+0000 7f50d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:02.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.274+0000 7f50d37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:02.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.274+0000 7f50d37fe640 10 monclient: dump: 2023-12-15T17:24:02.277 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:02.277 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:02.277 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.277 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.277 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:02.277 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.277 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50d37fe640 10 monclient: _finish_auth 0 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50d37fe640 10 monclient: _check_auth_tickets 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50d37fe640 1 -- 172.21.15.69:0/495963294 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e403e7b0 con 0x7f50ec1427c0 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50d37fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50d37fe640 1 -- 172.21.15.69:0/495963294 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e403eb60 con 0x7f50ec1427c0 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:02.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50d37fe640 10 monclient: dump: 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 5 monclient: authenticate success, global_id 21433 2023-12-15T17:24:02.279 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 10 monclient: _renew_subs 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 1 -- 172.21.15.69:0/495963294 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e4009a70 con 0x7f50ec1427c0 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50e97fa640 4 set_mon_vals no callback set 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 10 client.21433.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 10 monclient: _renew_subs 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 1 -- 172.21.15.69:0/495963294 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ec1384e0 con 0x7f50ec1427c0 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 1 librados: init done 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 10 client.21433.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 15 client.21433.objecter _session_command_op_assign -1 1 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 20 client.21433.objecter _get_session osd=-1 returning homeless 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 10 client.21433.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:02.280 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.275+0000 7f50ebfff640 1 -- 172.21.15.69:0/495963294 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f50ec0b2210 con 0x7f50ec1427c0 2023-12-15T17:24:02.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.295+0000 7f50d37fe640 1 -- 172.21.15.69:0/495963294 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f50e403b050 con 0x7f50ec1427c0 2023-12-15T17:24:02.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.295+0000 7f50d37fe640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f50d806cba0 0x7f50d806f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.295+0000 7f50eaffd640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f50d806cba0 0x7f50d806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.295+0000 7f50eaffd640 10 monclient: get_auth_request con 0x7f50d806cba0 auth_method 0 2023-12-15T17:24:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.296+0000 7f50d37fe640 10 client.21433.objecter ms_handle_connect 0x7f50d806cba0 2023-12-15T17:24:02.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.296+0000 7f50eaffd640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f50d806cba0 0x7f50d806f060 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009e20 tx=0x7f50dc009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 1 -- 172.21.15.69:0/495963294 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 534..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7f50e407b210 con 0x7f50ec1427c0 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 10 client.21433.objecter ms_dispatch 0x7f50ec000b90 osd_map(908..908 src has 534..908) v4 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 3 client.21433.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 3 client.21433.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 10 client.21433.objecter checking command 1 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] conn(0x7f50d8074430 0x7f50d80768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 20 client.21433.objecter _get_session s=0x7f50d8073d60 osd=0 3 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 20 client.21433.objecter put_session s=0x7f50d8073d60 osd=0 3 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 15 client.21433.objecter _session_command_op_remove -1 1 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 20 client.21433.objecter _get_session s=0x7f50d8073d60 osd=0 3 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 20 client.21433.objecter get_session s=0x7f50d8073d60 osd=0 3 2023-12-15T17:24:02.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 15 client.21433.objecter _session_command_op_assign 0 1 2023-12-15T17:24:02.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 20 client.21433.objecter put_session s=0x7f50d8073d60 osd=0 4 2023-12-15T17:24:02.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 10 client.21433.objecter _send_command 1 2023-12-15T17:24:02.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 1 -- 172.21.15.69:0/495963294 --> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50d8076fa0 con 0x7f50d8074430 2023-12-15T17:24:02.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 20 client.21433.objecter dump_active .. 0 homeless 2023-12-15T17:24:02.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 1 -- 172.21.15.69:0/495963294 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=908) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50e4077550 con 0x7f50ec1427c0 2023-12-15T17:24:02.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50d37fe640 10 monclient: handle_get_version_reply finishing 1 version 908 2023-12-15T17:24:02.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.298+0000 7f50ea7fc640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] conn(0x7f50d8074430 0x7f50d80768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.299+0000 7f50ea7fc640 10 monclient: get_auth_request con 0x7f50d8074430 auth_method 0 2023-12-15T17:24:02.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.301+0000 7f50d37fe640 10 client.21433.objecter ms_handle_connect 0x7f50d8074430 2023-12-15T17:24:02.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.301+0000 7f50ea7fc640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] conn(0x7f50d8074430 0x7f50d80768d0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:02.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.324+0000 7f50d37fe640 1 -- 172.21.15.69:0/495963294 <== osd.0 v2:172.21.15.69:6816/541068705 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f50d8076fa0 con 0x7f50d8074430 2023-12-15T17:24:02.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.324+0000 7f50d37fe640 10 client.21433.objecter ms_dispatch 0x7f50ec000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:24:02.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.324+0000 7f50d37fe640 10 client.21433.objecter _finish_command 1 = system:0 2023-12-15T17:24:02.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.324+0000 7f50d37fe640 20 client.21433.objecter put_session s=0x7f50d8073d60 osd=0 4 2023-12-15T17:24:02.326 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.324+0000 7f50d37fe640 15 client.21433.objecter _session_command_op_remove 0 1 2023-12-15T17:24:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.337+0000 7f50ebfff640 10 client.21433.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-15T17:24:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.337+0000 7f50ebfff640 15 client.21433.objecter _session_command_op_assign -1 2 2023-12-15T17:24:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.337+0000 7f50ebfff640 20 client.21433.objecter _get_session s=0x7f50d8073d60 osd=0 3 2023-12-15T17:24:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.337+0000 7f50ebfff640 20 client.21433.objecter put_session s=0x7f50d8073d60 osd=0 3 2023-12-15T17:24:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.337+0000 7f50ebfff640 20 client.21433.objecter _get_session s=0x7f50d8073d60 osd=0 3 2023-12-15T17:24:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.337+0000 7f50ebfff640 15 client.21433.objecter _session_command_op_remove -1 2 2023-12-15T17:24:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.337+0000 7f50ebfff640 20 client.21433.objecter get_session s=0x7f50d8073d60 osd=0 3 2023-12-15T17:24:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.337+0000 7f50ebfff640 15 client.21433.objecter _session_command_op_assign 0 2 2023-12-15T17:24:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.337+0000 7f50ebfff640 20 client.21433.objecter put_session s=0x7f50d8073d60 osd=0 4 2023-12-15T17:24:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.337+0000 7f50ebfff640 10 client.21433.objecter _send_command 2 2023-12-15T17:24:02.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.337+0000 7f50ebfff640 1 -- 172.21.15.69:0/495963294 --> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f50ec0a9e30 con 0x7f50d8074430 2023-12-15T17:24:02.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50d37fe640 1 -- 172.21.15.69:0/495963294 <== osd.0 v2:172.21.15.69:6816/541068705 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f50ec0a9e30 con 0x7f50d8074430 2023-12-15T17:24:02.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50d37fe640 10 client.21433.objecter ms_dispatch 0x7f50ec000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:24:02.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50d37fe640 10 client.21433.objecter _finish_command 2 = system:0 2023-12-15T17:24:02.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50d37fe640 20 client.21433.objecter put_session s=0x7f50d8073d60 osd=0 4 2023-12-15T17:24:02.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50d37fe640 15 client.21433.objecter _session_command_op_remove 0 2 2023-12-15T17:24:02.352 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-12-15T17:24:02.352 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-15T17:24:02.352 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 10 librados: watch_flush enter 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 10 librados: watch_flush exit 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 10 client.21433.objecter close_session for osd.0 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 1 -- 172.21.15.69:0/495963294 >> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] conn(0x7f50d8074430 msgr2=0x7f50d80768d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.69:6816/541068705,v1:172.21.15.69:6817/541068705] conn(0x7f50d8074430 0x7f50d80768d0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 20 client.21433.objecter put_session s=0x7f50d8073d60 osd=0 1 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 20 client.21433.objecter shutdown clearing up homeless session... 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 10 client.21433.objecter successfully canceled tick 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 1 -- 172.21.15.69:0/495963294 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f50d806cba0 msgr2=0x7f50d806f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f50d806cba0 0x7f50d806f060 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009e20 tx=0x7f50dc009240 comp rx=0 tx=0).stop 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 10 monclient: shutdown 2023-12-15T17:24:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 1 -- 172.21.15.69:0/495963294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50ec1427c0 msgr2=0x7f50ec1da7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50ebfff640 1 --2- 172.21.15.69:0/495963294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50ec1427c0 0x7f50ec1da7d0 secure :-1 s=READY pgs=5104 cs=0 l=1 rev1=1 crypto rx=0x7f50e4033000 tx=0x7f50e4004a40 comp rx=0 tx=0).stop 2023-12-15T17:24:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.350+0000 7f50eb7fe640 1 -- 172.21.15.69:0/495963294 reap_dead start 2023-12-15T17:24:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.351+0000 7f50ebfff640 1 -- 172.21.15.69:0/495963294 shutdown_connections 2023-12-15T17:24:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.351+0000 7f50ebfff640 1 -- 172.21.15.69:0/495963294 >> 172.21.15.69:0/495963294 conn(0x7f50ec0617a0 msgr2=0x7f50ec0b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.351+0000 7f50ebfff640 1 -- 172.21.15.69:0/495963294 shutdown_connections 2023-12-15T17:24:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.351+0000 7f50ebfff640 1 -- 172.21.15.69:0/495963294 wait complete. 2023-12-15T17:24:02.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.351+0000 7f50ebfff640 1 librados: shutdown 2023-12-15T17:24:02.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-15T17:24:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.464+0000 7fbb50b33640 10 monclient: get_monmap_and_config 2023-12-15T17:24:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.464+0000 7fbb50b33640 10 monclient: build_initial_monmap 2023-12-15T17:24:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.464+0000 7fbb50b33640 10 monclient: monmap: 2023-12-15T17:24:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:02.465518+0000 2023-12-15T17:24:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:02.465518+0000 2023-12-15T17:24:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:02.466 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.464+0000 7fbb50b33640 1 Processor -- start 2023-12-15T17:24:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.464+0000 7fbb50b33640 1 -- start start 2023-12-15T17:24:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.464+0000 7fbb50b33640 10 monclient: init 2023-12-15T17:24:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-12-15T17:24:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbb4c0a8910 0x7fbb4c13ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 10 monclient(hunting): picked mon.noname-c con 0x7fbb4c0a8910 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb4c0a9b70 0x7fbb4c13f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 10 monclient(hunting): picked mon.noname-b con 0x7fbb4c0a9b70 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb4c141f90 addr v1:172.21.15.69:6789/0 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4c0a8cf0 con 0x7fbb4c141f90 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4c13f6e0 con 0x7fbb4c0a8910 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c0aabd0 con 0x7fbb4c0a9b70 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb50b33640 10 monclient(hunting): authenticate will time out at 2746.781801s 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.465+0000 7fbb49d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb4c0a9b70 0x7fbb4c13f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb4ad9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbb4c141f90 0x7fbb4c142400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42680/0 (socket says 172.21.15.69:42680) 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb4ad9d640 1 -- 172.21.15.69:0/3183566724 learned_addr learned my addr 172.21.15.69:0/3183566724 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb49d9b640 10 monclient(hunting): get_auth_request con 0x7fbb4c0a9b70 auth_method 0 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb49d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb49d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb4a59c640 1 --2- 172.21.15.69:0/3183566724 >> v2:172.21.15.192:3300/0 conn(0x7fbb4c0a8910 0x7fbb4c13ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb4a59c640 10 monclient(hunting): get_auth_request con 0x7fbb4c0a8910 auth_method 0 2023-12-15T17:24:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb4a59c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb4a59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.466+0000 7fbb49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.467+0000 7fbb49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.467+0000 7fbb4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.467+0000 7fbb4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.467+0000 7fbb4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.468+0000 7fbb4a59c640 10 monclient(hunting): handle_auth_done global_id 21436 payload 274 2023-12-15T17:24:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.468+0000 7fbb4a59c640 1 -- 172.21.15.69:0/3183566724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb4c0a9b70 msgr2=0x7fbb4c13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.468+0000 7fbb4a59c640 1 --2- 172.21.15.69:0/3183566724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb4c0a9b70 0x7fbb4c13f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.468+0000 7fbb4a59c640 1 -- 172.21.15.69:0/3183566724 >> v1:172.21.15.69:6789/0 conn(0x7fbb4c141f90 legacy=0x7fbb4c142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.468+0000 7fbb4a59c640 10 monclient: _finish_hunting 0 2023-12-15T17:24:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.468+0000 7fbb4a59c640 1 monclient: found mon.noname-c 2023-12-15T17:24:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.468+0000 7fbb4a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.468+0000 7fbb4a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:24:02.471 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.468+0000 7fbb4a59c640 1 -- 172.21.15.69:0/3183566724 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c0aaeb0 con 0x7fbb4c0a8910 2023-12-15T17:24:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.471+0000 7fbb4a59c640 1 --2- 172.21.15.69:0/3183566724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c0a8910 0x7fbb4c13ec60 secure :-1 s=READY pgs=5105 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009d90 tx=0x7fbb4402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3783a8f56b8533b5 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 1 -- 172.21.15.69:0/3183566724 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4402cc50 con 0x7fbb4c0a8910 2023-12-15T17:24:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 10 monclient: dump: 2023-12-15T17:24:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.477 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 10 monclient: _finish_auth 0 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 10 monclient: _check_auth_tickets 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb50b33640 5 monclient: authenticate success, global_id 21436 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 1 -- 172.21.15.69:0/3183566724 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4402cdb0 con 0x7fbb4c0a8910 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb50b33640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb50b33640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb50b33640 4 set_mon_vals no callback set 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 1 -- 172.21.15.69:0/3183566724 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44034660 con 0x7fbb4c0a8910 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:02.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.474+0000 7fbb4959a640 10 monclient: dump: 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.475+0000 7fbb50b33640 10 monclient: shutdown 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.475+0000 7fbb50b33640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.475+0000 7fbb50b33640 1 -- 172.21.15.69:0/3183566724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c0a8910 msgr2=0x7fbb4c13ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.475+0000 7fbb50b33640 1 --2- 172.21.15.69:0/3183566724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c0a8910 0x7fbb4c13ec60 secure :-1 s=READY pgs=5105 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009d90 tx=0x7fbb4402bea0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.475+0000 7fbb50b33640 1 -- 172.21.15.69:0/3183566724 shutdown_connections 2023-12-15T17:24:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.475+0000 7fbb50b33640 1 --2- 172.21.15.69:0/3183566724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb4c0a9b70 0x7fbb4c13f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.475+0000 7fbb50b33640 1 --2- 172.21.15.69:0/3183566724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c0a8910 0x7fbb4c13ec60 unknown :-1 s=CLOSED pgs=5105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.475+0000 7fbb50b33640 1 -- 172.21.15.69:0/3183566724 >> 172.21.15.69:0/3183566724 conn(0x7fbb4c0b7780 msgr2=0x7fbb4c0b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.475+0000 7fbb50b33640 1 -- 172.21.15.69:0/3183566724 shutdown_connections 2023-12-15T17:24:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.475+0000 7fbb50b33640 1 -- 172.21.15.69:0/3183566724 wait complete. 2023-12-15T17:24:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.476+0000 7fbb50b33640 10 monclient: build_initial_monmap 2023-12-15T17:24:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.476+0000 7fbb50b33640 10 monclient: monmap: 2023-12-15T17:24:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:02.483 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.476+0000 7fbb50b33640 1 librados: starting msgr at 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.476+0000 7fbb50b33640 1 librados: starting objecter 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.476+0000 7fbb50b33640 1 Processor -- start 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.476+0000 7fbb50b33640 1 -- start start 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.476+0000 7fbb50b33640 1 librados: setting wanted keys 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.476+0000 7fbb50b33640 1 librados: calling monclient init 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.476+0000 7fbb50b33640 10 monclient: init 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb4c0a8910 0x7fbb4c137350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 10 monclient(hunting): picked mon.noname-c con 0x7fbb4c0a8910 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb4c141f90 0x7fbb4c1d6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 10 monclient(hunting): picked mon.noname-a con 0x7fbb4c141f90 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:02.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c1d7270 0x7fbb4c0b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 10 monclient(hunting): picked mon.noname-b con 0x7fbb4c1d7270 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c1d7720 con 0x7fbb4c141f90 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c1d79a0 con 0x7fbb4c1d7270 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c0b0cc0 con 0x7fbb4c0a8910 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb50b33640 10 monclient(hunting): authenticate will time out at 2746.793792s 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.477+0000 7fbb49d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb4c141f90 0x7fbb4c1d6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4a59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb4c0a8910 0x7fbb4c137350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4ad9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c1d7270 0x7fbb4c0b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb49d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb4c141f90 0x7fbb4c1d6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45356/0 (socket says 172.21.15.69:45356) 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb49d9b640 1 -- 172.21.15.69:0/1441565497 learned_addr learned my addr 172.21.15.69:0/1441565497 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:02.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb49d9b640 10 monclient(hunting): get_auth_request con 0x7fbb4c141f90 auth_method 0 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb49d9b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb49d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4ad9d640 10 monclient(hunting): get_auth_request con 0x7fbb4c1d7270 auth_method 0 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4ad9d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4a59c640 10 monclient(hunting): get_auth_request con 0x7fbb4c0a8910 auth_method 0 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4a59c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4a59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:02.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.478+0000 7fbb49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.481+0000 7fbb4ad9d640 10 monclient(hunting): handle_auth_done global_id 21439 payload 824 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.481+0000 7fbb4ad9d640 1 -- 172.21.15.69:0/1441565497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb4c0a8910 msgr2=0x7fbb4c137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.481+0000 7fbb4ad9d640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb4c0a8910 0x7fbb4c137350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.481+0000 7fbb4ad9d640 1 -- 172.21.15.69:0/1441565497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb4c141f90 msgr2=0x7fbb4c1d6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.481+0000 7fbb4ad9d640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb4c141f90 0x7fbb4c1d6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.481+0000 7fbb4ad9d640 10 monclient: _finish_hunting 0 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.481+0000 7fbb4ad9d640 1 monclient: found mon.noname-b 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.481+0000 7fbb4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.481+0000 7fbb4ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:02.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.481+0000 7fbb4ad9d640 1 -- 172.21.15.69:0/1441565497 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb44009a70 con 0x7fbb4c1d7270 2023-12-15T17:24:02.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.484+0000 7fbb4ad9d640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c1d7270 0x7fbb4c0b0780 secure :-1 s=READY pgs=5106 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00c3c0 tx=0x7fbb3c00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:02.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.484+0000 7fbb42ffd640 10 client.?.objecter ms_handle_connect 0x7fbb4c1d7270 2023-12-15T17:24:02.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.484+0000 7fbb42ffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 1 -- 172.21.15.69:0/1441565497 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c005ce0 con 0x7fbb4c1d7270 2023-12-15T17:24:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:02.489 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 10 monclient: dump: 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 10 monclient: _finish_auth 0 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:02.490 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 1 -- 172.21.15.69:0/1441565497 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c005e40 con 0x7fbb4c1d7270 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb50b33640 5 monclient: authenticate success, global_id 21439 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 1 -- 172.21.15.69:0/1441565497 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c0072f0 con 0x7fbb4c1d7270 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb50b33640 10 monclient: _renew_subs 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb50b33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb50b33640 1 -- 172.21.15.69:0/1441565497 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb4c0b0f40 con 0x7fbb4c1d7270 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb48d99640 4 set_mon_vals no callback set 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb42ffd640 10 monclient: dump: 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:02.491 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb50b33640 10 client.21439.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb50b33640 10 monclient: _renew_subs 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb50b33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb50b33640 1 -- 172.21.15.69:0/1441565497 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4c13a7b0 con 0x7fbb4c1d7270 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.487+0000 7fbb50b33640 1 librados: init done 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.488+0000 7fbb50b33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.488+0000 7fbb50b33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.488+0000 7fbb50b33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:02.492 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.488+0000 7fbb50b33640 1 -- 172.21.15.69:0/1441565497 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb4c075b60 con 0x7fbb4c1d7270 2023-12-15T17:24:02.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.507+0000 7fbb42ffd640 1 -- 172.21.15.69:0/1441565497 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fbb3c007450 con 0x7fbb4c1d7270 2023-12-15T17:24:02.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.507+0000 7fbb42ffd640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbb2c06cba0 0x7fbb2c06f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.507+0000 7fbb4a59c640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbb2c06cba0 0x7fbb2c06f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.507+0000 7fbb4a59c640 10 monclient: get_auth_request con 0x7fbb2c06cba0 auth_method 0 2023-12-15T17:24:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.508+0000 7fbb42ffd640 10 client.21439.objecter ms_handle_connect 0x7fbb2c06cba0 2023-12-15T17:24:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.508+0000 7fbb4a59c640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbb2c06cba0 0x7fbb2c06f060 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbb44036000 tx=0x7fbb44030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:02.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.510+0000 7fbb42ffd640 1 -- 172.21.15.69:0/1441565497 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 534..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7fbb3c058030 con 0x7fbb4c1d7270 2023-12-15T17:24:02.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.510+0000 7fbb42ffd640 10 client.21439.objecter ms_dispatch 0x7fbb4c000b90 osd_map(908..908 src has 534..908) v4 2023-12-15T17:24:02.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.510+0000 7fbb42ffd640 3 client.21439.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:02.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.510+0000 7fbb42ffd640 3 client.21439.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:02.512 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.510+0000 7fbb42ffd640 20 client.21439.objecter dump_active .. 0 homeless 2023-12-15T17:24:02.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.743+0000 7fbb42ffd640 1 -- 172.21.15.69:0/1441565497 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fbb3c054370 con 0x7fbb4c1d7270 2023-12-15T17:24:02.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.743+0000 7fbb42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:02.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.743+0000 7fbb42ffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:02.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.918+0000 7fbb50b33640 1 -- 172.21.15.69:0/1441565497 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb4c0a8fa0 con 0x7fbb2c06cba0 2023-12-15T17:24:02.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.918+0000 7fbb42ffd640 1 -- 172.21.15.69:0/1441565497 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbb44043070 con 0x7fbb2c06cba0 2023-12-15T17:24:02.920 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 0 to deep-scrub 2023-12-15T17:24:02.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 10 librados: watch_flush enter 2023-12-15T17:24:02.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 10 librados: watch_flush exit 2023-12-15T17:24:02.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 20 client.21439.objecter shutdown clearing up homeless session... 2023-12-15T17:24:02.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 10 client.21439.objecter successfully canceled tick 2023-12-15T17:24:02.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 -- 172.21.15.69:0/1441565497 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbb2c06cba0 msgr2=0x7fbb2c06f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbb2c06cba0 0x7fbb2c06f060 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbb44036000 tx=0x7fbb44030000 comp rx=0 tx=0).stop 2023-12-15T17:24:02.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 10 monclient: shutdown 2023-12-15T17:24:02.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:02.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 -- 172.21.15.69:0/1441565497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c1d7270 msgr2=0x7fbb4c0b0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:02.924 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c1d7270 0x7fbb4c0b0780 secure :-1 s=READY pgs=5106 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00c3c0 tx=0x7fbb3c00c880 comp rx=0 tx=0).stop 2023-12-15T17:24:02.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 -- 172.21.15.69:0/1441565497 shutdown_connections 2023-12-15T17:24:02.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fbb2c06cba0 0x7fbb2c06f060 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c1d7270 0x7fbb4c0b0780 unknown :-1 s=CLOSED pgs=5106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb4c141f90 0x7fbb4c1d6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 --2- 172.21.15.69:0/1441565497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb4c0a8910 0x7fbb4c137350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:02.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 -- 172.21.15.69:0/1441565497 >> 172.21.15.69:0/1441565497 conn(0x7fbb4c0b7780 msgr2=0x7fbb4c0b7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:02.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 -- 172.21.15.69:0/1441565497 shutdown_connections 2023-12-15T17:24:02.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 -- 172.21.15.69:0/1441565497 wait complete. 2023-12-15T17:24:02.925 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:02.922+0000 7fbb50b33640 1 librados: shutdown 2023-12-15T17:24:02.937 INFO:tasks.ceph:Scrubbing osd.1 2023-12-15T17:24:02.938 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-15T17:24:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.039+0000 7f7ac26d4640 10 monclient: get_monmap_and_config 2023-12-15T17:24:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.039+0000 7f7ac26d4640 10 monclient: build_initial_monmap 2023-12-15T17:24:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.039+0000 7f7ac26d4640 10 monclient: monmap: 2023-12-15T17:24:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:03.041115+0000 2023-12-15T17:24:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:03.041115+0000 2023-12-15T17:24:03.041 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:03.042 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:03.042 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 1 Processor -- start 2023-12-15T17:24:03.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 1 -- start start 2023-12-15T17:24:03.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 10 monclient: init 2023-12-15T17:24:03.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:03.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-12-15T17:24:03.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7abc071110 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:03.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 0x7f7abc0afc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 10 monclient(hunting): picked mon.noname-b con 0x7f7abc13f820 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:03.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7abc070470 0x7f7abc06c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.044 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 10 monclient(hunting): picked mon.noname-a con 0x7f7abc070470 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7abc06bf40 con 0x7f7abc071110 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7abc0714f0 con 0x7f7abc070470 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc071c30 con 0x7f7abc13f820 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac26d4640 10 monclient(hunting): authenticate will time out at 2747.357421s 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7abb7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 0x7f7abc0afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7ac0c71640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7abc070470 0x7f7abc06c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7abbfff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f7abc071110 0x7f7abc13f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:45448/0 (socket says 172.21.15.69:45448) 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.041+0000 7f7abbfff640 1 -- 172.21.15.69:0/2044732276 learned_addr learned my addr 172.21.15.69:0/2044732276 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7abb7fe640 10 monclient(hunting): get_auth_request con 0x7f7abc13f820 auth_method 0 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7abb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7abb7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7abb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7ac0c71640 10 monclient(hunting): get_auth_request con 0x7f7abc070470 auth_method 0 2023-12-15T17:24:03.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7ac0c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7ac0c71640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7ac0c71640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7abb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7abb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7abb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7ac0c71640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7ac0c71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.042+0000 7f7ac0c71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.044+0000 7f7abb7fe640 10 monclient(hunting): handle_auth_done global_id 17789 payload 274 2023-12-15T17:24:03.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.044+0000 7f7abb7fe640 1 -- 172.21.15.69:0/2044732276 >> v2:172.21.15.69:3300/0 conn(0x7f7abc070470 msgr2=0x7f7abc06c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:03.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.044+0000 7f7abb7fe640 1 --2- 172.21.15.69:0/2044732276 >> v2:172.21.15.69:3300/0 conn(0x7f7abc070470 0x7f7abc06c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.044+0000 7f7abb7fe640 1 -- 172.21.15.69:0/2044732276 >> v1:172.21.15.192:6789/0 conn(0x7f7abc071110 legacy=0x7f7abc13f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.044+0000 7f7abb7fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:03.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.044+0000 7f7abb7fe640 1 monclient: found mon.noname-b 2023-12-15T17:24:03.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.044+0000 7f7abb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:03.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.044+0000 7f7abb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:24:03.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.044+0000 7f7abb7fe640 1 -- 172.21.15.69:0/2044732276 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc071710 con 0x7f7abc13f820 2023-12-15T17:24:03.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.046+0000 7f7abb7fe640 1 --2- 172.21.15.69:0/2044732276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 0x7f7abc0afc10 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00097a0 tx=0x7f7ab002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2978286a79952c2 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 1 -- 172.21.15.69:0/2044732276 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab002e070 con 0x7f7abc13f820 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 10 monclient: dump: 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:03.051 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 10 monclient: _finish_auth 0 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 10 monclient: _check_auth_tickets 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 1 -- 172.21.15.69:0/2044732276 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab00315a0 con 0x7f7abc13f820 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 1 -- 172.21.15.69:0/2044732276 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0031930 con 0x7f7abc13f820 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 5 monclient: authenticate success, global_id 17789 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 4 set_mon_vals no callback set 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7abaffd640 10 monclient: dump: 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.052 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.053 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:03.053 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.053 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:03.053 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:03.053 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:03.053 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 10 monclient: shutdown 2023-12-15T17:24:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 1 -- 172.21.15.69:0/2044732276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 msgr2=0x7f7abc0afc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 1 --2- 172.21.15.69:0/2044732276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 0x7f7abc0afc10 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00097a0 tx=0x7f7ab002b840 comp rx=0 tx=0).stop 2023-12-15T17:24:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 1 -- 172.21.15.69:0/2044732276 shutdown_connections 2023-12-15T17:24:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 1 --2- 172.21.15.69:0/2044732276 >> v2:172.21.15.69:3300/0 conn(0x7f7abc070470 0x7f7abc06c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 1 --2- 172.21.15.69:0/2044732276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 0x7f7abc0afc10 unknown :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 1 -- 172.21.15.69:0/2044732276 >> 172.21.15.69:0/2044732276 conn(0x7f7abc0b5ca0 msgr2=0x7f7abc0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 1 -- 172.21.15.69:0/2044732276 shutdown_connections 2023-12-15T17:24:03.056 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.050+0000 7f7ac26d4640 1 -- 172.21.15.69:0/2044732276 wait complete. 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.051+0000 7f7ac26d4640 10 monclient: build_initial_monmap 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.051+0000 7f7ac26d4640 10 monclient: monmap: 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.051+0000 7f7ac26d4640 1 librados: starting msgr at 2023-12-15T17:24:03.057 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.051+0000 7f7ac26d4640 1 librados: starting objecter 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.051+0000 7f7ac26d4640 1 Processor -- start 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.051+0000 7f7ac26d4640 1 -- start start 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.051+0000 7f7ac26d4640 1 librados: setting wanted keys 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.051+0000 7f7ac26d4640 1 librados: calling monclient init 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.051+0000 7f7ac26d4640 10 monclient: init 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7abc070470 0x7f7abc138160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 10 monclient(hunting): picked mon.noname-b con 0x7f7abc070470 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7abc071110 0x7f7abc141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 10 monclient(hunting): picked mon.noname-a con 0x7f7abc071110 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 0x7f7abc141650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 10 monclient(hunting): picked mon.noname-c con 0x7f7abc13f820 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.058 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc139180 con 0x7f7abc071110 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc138e50 con 0x7f7abc070470 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc140a10 con 0x7f7abc13f820 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac26d4640 10 monclient(hunting): authenticate will time out at 2747.368589s 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7abb7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7abc071110 0x7f7abc141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.052+0000 7f7ac0c71640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 0x7f7abc141650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7abc070470 0x7f7abc138160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7ac0c71640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 0x7f7abc141650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42354/0 (socket says 172.21.15.69:42354) 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7ac0c71640 1 -- 172.21.15.69:0/1403402874 learned_addr learned my addr 172.21.15.69:0/1403402874 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7ac0c71640 10 monclient(hunting): get_auth_request con 0x7f7abc13f820 auth_method 0 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abb7fe640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7abc071110 0x7f7abc141110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45366/0 (socket says 172.21.15.69:45366) 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7ac0c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.059 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7ac0c71640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7ac0c71640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abb7fe640 10 monclient(hunting): get_auth_request con 0x7f7abc071110 auth_method 0 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abb7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abbfff640 10 monclient(hunting): get_auth_request con 0x7f7abc070470 auth_method 0 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abbfff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7ac0c71640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7ac0c71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7ac0c71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.060 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.053+0000 7f7abbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.061 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.056+0000 7f7ac0c71640 10 monclient(hunting): handle_auth_done global_id 17792 payload 824 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.056+0000 7f7ac0c71640 1 -- 172.21.15.69:0/1403402874 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7abc070470 msgr2=0x7f7abc138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.056+0000 7f7ac0c71640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7abc070470 0x7f7abc138160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.056+0000 7f7ac0c71640 1 -- 172.21.15.69:0/1403402874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7abc071110 msgr2=0x7f7abc141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.056+0000 7f7ac0c71640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7abc071110 0x7f7abc141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.056+0000 7f7ac0c71640 10 monclient: _finish_hunting 0 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.056+0000 7f7ac0c71640 1 monclient: found mon.noname-c 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.056+0000 7f7ac0c71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.056+0000 7f7ac0c71640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.056+0000 7f7ac0c71640 1 -- 172.21.15.69:0/1403402874 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab0009400 con 0x7f7abc13f820 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.058+0000 7f7ac0c71640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 0x7f7abc141650 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400ef90 tx=0x7f7ab400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.058+0000 7f7ab8ff9640 10 client.?.objecter ms_handle_connect 0x7f7abc13f820 2023-12-15T17:24:03.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.058+0000 7f7ab8ff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 1 -- 172.21.15.69:0/1403402874 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab4009bf0 con 0x7f7abc13f820 2023-12-15T17:24:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 10 monclient: dump: 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 10 monclient: _finish_auth 0 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 10 monclient: _check_auth_tickets 2023-12-15T17:24:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:03.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 1 -- 172.21.15.69:0/1403402874 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab400f040 con 0x7f7abc13f820 2023-12-15T17:24:03.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:03.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 1 -- 172.21.15.69:0/1403402874 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab40146c0 con 0x7f7abc13f820 2023-12-15T17:24:03.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:03.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:03.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ab8ff9640 10 monclient: dump: 2023-12-15T17:24:03.065 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:03.065 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.065 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.065 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.065 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:03.066 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ac26d4640 5 monclient: authenticate success, global_id 17792 2023-12-15T17:24:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ac26d4640 10 monclient: _renew_subs 2023-12-15T17:24:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ac26d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ac26d4640 1 -- 172.21.15.69:0/1403402874 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7abc140cf0 con 0x7f7abc13f820 2023-12-15T17:24:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7aba7fc640 4 set_mon_vals no callback set 2023-12-15T17:24:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ac26d4640 10 client.17792.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ac26d4640 10 monclient: _renew_subs 2023-12-15T17:24:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ac26d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ac26d4640 1 -- 172.21.15.69:0/1403402874 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7abc1386a0 con 0x7f7abc13f820 2023-12-15T17:24:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.062+0000 7f7ac26d4640 1 librados: init done 2023-12-15T17:24:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.063+0000 7f7ac26d4640 10 client.17792.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.063+0000 7f7ac26d4640 15 client.17792.objecter _session_command_op_assign -1 1 2023-12-15T17:24:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.063+0000 7f7ac26d4640 20 client.17792.objecter _get_session osd=-1 returning homeless 2023-12-15T17:24:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.063+0000 7f7ac26d4640 10 client.17792.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.063+0000 7f7ac26d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.063+0000 7f7ac26d4640 1 -- 172.21.15.69:0/1403402874 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7abc1392f0 con 0x7f7abc13f820 2023-12-15T17:24:03.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.083+0000 7f7ab8ff9640 1 -- 172.21.15.69:0/1403402874 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7ab4014820 con 0x7f7abc13f820 2023-12-15T17:24:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.083+0000 7f7ab8ff9640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7aa806cb50 0x7f7aa806f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.084+0000 7f7abbfff640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7aa806cb50 0x7f7aa806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.084+0000 7f7abbfff640 10 monclient: get_auth_request con 0x7f7aa806cb50 auth_method 0 2023-12-15T17:24:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.084+0000 7f7ab8ff9640 10 client.17792.objecter ms_handle_connect 0x7f7aa806cb50 2023-12-15T17:24:03.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.084+0000 7f7abbfff640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7aa806cb50 0x7f7aa806f010 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7aac005880 tx=0x7f7aac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:03.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 1 -- 172.21.15.69:0/1403402874 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(908..908 src has 534..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7f7ab4059ba0 con 0x7f7abc13f820 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 10 client.17792.objecter ms_dispatch 0x7f7abc000b90 osd_map(908..908 src has 534..908) v4 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 3 client.17792.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 3 client.17792.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 10 client.17792.objecter checking command 1 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] conn(0x7f7aa80743e0 0x7f7aa8076880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 20 client.17792.objecter _get_session s=0x7f7aa8073d10 osd=1 3 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 20 client.17792.objecter put_session s=0x7f7aa8073d10 osd=1 3 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 15 client.17792.objecter _session_command_op_remove -1 1 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 20 client.17792.objecter _get_session s=0x7f7aa8073d10 osd=1 3 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 20 client.17792.objecter get_session s=0x7f7aa8073d10 osd=1 3 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 15 client.17792.objecter _session_command_op_assign 1 1 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 20 client.17792.objecter put_session s=0x7f7aa8073d10 osd=1 4 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 10 client.17792.objecter _send_command 1 2023-12-15T17:24:03.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 1 -- 172.21.15.69:0/1403402874 --> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7aa8076f50 con 0x7f7aa80743e0 2023-12-15T17:24:03.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 20 client.17792.objecter dump_active .. 0 homeless 2023-12-15T17:24:03.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 1 -- 172.21.15.69:0/1403402874 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=908) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ab4055ee0 con 0x7f7abc13f820 2023-12-15T17:24:03.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.087+0000 7f7ab8ff9640 10 monclient: handle_get_version_reply finishing 1 version 908 2023-12-15T17:24:03.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.088+0000 7f7abb7fe640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] conn(0x7f7aa80743e0 0x7f7aa8076880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.088+0000 7f7abb7fe640 10 monclient: get_auth_request con 0x7f7aa80743e0 auth_method 0 2023-12-15T17:24:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.091+0000 7f7ab8ff9640 10 client.17792.objecter ms_handle_connect 0x7f7aa80743e0 2023-12-15T17:24:03.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.091+0000 7f7abb7fe640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] conn(0x7f7aa80743e0 0x7f7aa8076880 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:03.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.114+0000 7f7ab8ff9640 1 -- 172.21.15.69:0/1403402874 <== osd.1 v2:172.21.15.69:6808/1683202761 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7aa8076f50 con 0x7f7aa80743e0 2023-12-15T17:24:03.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.114+0000 7f7ab8ff9640 10 client.17792.objecter ms_dispatch 0x7f7abc000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:24:03.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.114+0000 7f7ab8ff9640 10 client.17792.objecter _finish_command 1 = system:0 2023-12-15T17:24:03.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.114+0000 7f7ab8ff9640 20 client.17792.objecter put_session s=0x7f7aa8073d10 osd=1 4 2023-12-15T17:24:03.116 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.114+0000 7f7ab8ff9640 15 client.17792.objecter _session_command_op_remove 1 1 2023-12-15T17:24:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.127+0000 7f7ac26d4640 10 client.17792.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-15T17:24:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.127+0000 7f7ac26d4640 15 client.17792.objecter _session_command_op_assign -1 2 2023-12-15T17:24:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.127+0000 7f7ac26d4640 20 client.17792.objecter _get_session s=0x7f7aa8073d10 osd=1 3 2023-12-15T17:24:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.127+0000 7f7ac26d4640 20 client.17792.objecter put_session s=0x7f7aa8073d10 osd=1 3 2023-12-15T17:24:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.127+0000 7f7ac26d4640 20 client.17792.objecter _get_session s=0x7f7aa8073d10 osd=1 3 2023-12-15T17:24:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.127+0000 7f7ac26d4640 15 client.17792.objecter _session_command_op_remove -1 2 2023-12-15T17:24:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.127+0000 7f7ac26d4640 20 client.17792.objecter get_session s=0x7f7aa8073d10 osd=1 3 2023-12-15T17:24:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.127+0000 7f7ac26d4640 15 client.17792.objecter _session_command_op_assign 1 2 2023-12-15T17:24:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.127+0000 7f7ac26d4640 20 client.17792.objecter put_session s=0x7f7aa8073d10 osd=1 4 2023-12-15T17:24:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.127+0000 7f7ac26d4640 10 client.17792.objecter _send_command 2 2023-12-15T17:24:03.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.127+0000 7f7ac26d4640 1 -- 172.21.15.69:0/1403402874 --> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7abc075af0 con 0x7f7aa80743e0 2023-12-15T17:24:03.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.140+0000 7f7ab8ff9640 1 -- 172.21.15.69:0/1403402874 <== osd.1 v2:172.21.15.69:6808/1683202761 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7abc075af0 con 0x7f7aa80743e0 2023-12-15T17:24:03.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.140+0000 7f7ab8ff9640 10 client.17792.objecter ms_dispatch 0x7f7abc000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:24:03.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.140+0000 7f7ab8ff9640 10 client.17792.objecter _finish_command 2 = system:0 2023-12-15T17:24:03.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.140+0000 7f7ab8ff9640 20 client.17792.objecter put_session s=0x7f7aa8073d10 osd=1 4 2023-12-15T17:24:03.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.140+0000 7f7ab8ff9640 15 client.17792.objecter _session_command_op_remove 1 2 2023-12-15T17:24:03.142 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-12-15T17:24:03.143 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-15T17:24:03.143 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-12-15T17:24:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 10 librados: watch_flush enter 2023-12-15T17:24:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 10 librados: watch_flush exit 2023-12-15T17:24:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 10 client.17792.objecter close_session for osd.1 2023-12-15T17:24:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 1 -- 172.21.15.69:0/1403402874 >> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] conn(0x7f7aa80743e0 msgr2=0x7f7aa8076880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.69:6808/1683202761,v1:172.21.15.69:6809/1683202761] conn(0x7f7aa80743e0 0x7f7aa8076880 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 20 client.17792.objecter put_session s=0x7f7aa8073d10 osd=1 1 2023-12-15T17:24:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 20 client.17792.objecter shutdown clearing up homeless session... 2023-12-15T17:24:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 10 client.17792.objecter successfully canceled tick 2023-12-15T17:24:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 1 -- 172.21.15.69:0/1403402874 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7aa806cb50 msgr2=0x7f7aa806f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7aa806cb50 0x7f7aa806f010 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7aac005880 tx=0x7f7aac009040 comp rx=0 tx=0).stop 2023-12-15T17:24:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 10 monclient: shutdown 2023-12-15T17:24:03.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:03.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 1 -- 172.21.15.69:0/1403402874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 msgr2=0x7f7abc141650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 1 --2- 172.21.15.69:0/1403402874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7abc13f820 0x7f7abc141650 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400ef90 tx=0x7f7ab400c490 comp rx=0 tx=0).stop 2023-12-15T17:24:03.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac0c71640 1 -- 172.21.15.69:0/1403402874 reap_dead start 2023-12-15T17:24:03.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 1 -- 172.21.15.69:0/1403402874 shutdown_connections 2023-12-15T17:24:03.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 1 -- 172.21.15.69:0/1403402874 >> 172.21.15.69:0/1403402874 conn(0x7f7abc0b5ca0 msgr2=0x7f7abc0b3380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:03.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 1 -- 172.21.15.69:0/1403402874 shutdown_connections 2023-12-15T17:24:03.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 1 -- 172.21.15.69:0/1403402874 wait complete. 2023-12-15T17:24:03.145 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.141+0000 7f7ac26d4640 1 librados: shutdown 2023-12-15T17:24:03.153 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-15T17:24:03.251 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.251+0000 7f2645cc9640 10 monclient: get_monmap_and_config 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.251+0000 7f2645cc9640 10 monclient: build_initial_monmap 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.251+0000 7f2645cc9640 10 monclient: monmap: 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:03.251860+0000 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:03.251860+0000 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:03.252 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 1 Processor -- start 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 1 -- start start 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 10 monclient: init 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f26400af0f0 0x7f26400ad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f26400af0f0 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400af4d0 0x7f2640142040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 10 monclient(hunting): picked mon.noname-b con 0x7f26400af4d0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26400adaa0 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2640072340 con 0x7f26400adaa0 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2640070b90 con 0x7f26400af0f0 2023-12-15T17:24:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26400afe80 con 0x7f26400af4d0 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.252+0000 7f2645cc9640 10 monclient(hunting): authenticate will time out at 2747.568127s 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263f7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f26400af0f0 0x7f26400ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400af4d0 0x7f2640142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400af4d0 0x7f2640142040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42362/0 (socket says 172.21.15.69:42362) 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263effd640 1 -- 172.21.15.69:0/3579831964 learned_addr learned my addr 172.21.15.69:0/3579831964 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263effd640 10 monclient(hunting): get_auth_request con 0x7f26400af4d0 auth_method 0 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263effd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263f7fe640 10 monclient(hunting): get_auth_request con 0x7f26400af0f0 auth_method 0 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.256 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.253+0000 7f263f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.254+0000 7f263effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.254+0000 7f263effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.254+0000 7f263effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.254+0000 7f263f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.254+0000 7f263f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.257 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.254+0000 7f263f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.255+0000 7f263effd640 10 monclient(hunting): handle_auth_done global_id 17795 payload 274 2023-12-15T17:24:03.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.255+0000 7f263effd640 1 -- 172.21.15.69:0/3579831964 >> v2:172.21.15.192:3300/0 conn(0x7f26400af0f0 msgr2=0x7f26400ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:03.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.255+0000 7f263effd640 1 --2- 172.21.15.69:0/3579831964 >> v2:172.21.15.192:3300/0 conn(0x7f26400af0f0 0x7f26400ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.255+0000 7f263effd640 1 -- 172.21.15.69:0/3579831964 >> v1:172.21.15.192:6789/0 conn(0x7f26400adaa0 legacy=0x7f264006f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.255+0000 7f263effd640 10 monclient: _finish_hunting 0 2023-12-15T17:24:03.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.255+0000 7f263effd640 1 monclient: found mon.noname-b 2023-12-15T17:24:03.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.256+0000 7f263effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:03.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.256+0000 7f263effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:24:03.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.256+0000 7f263effd640 1 -- 172.21.15.69:0/3579831964 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264013c600 con 0x7f26400af4d0 2023-12-15T17:24:03.258 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.257+0000 7f263effd640 1 --2- 172.21.15.69:0/3579831964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400af4d0 0x7f2640142040 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 crypto rx=0x7f2634009870 tx=0x7f263402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cb2906d4b873e3f server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 1 -- 172.21.15.69:0/3579831964 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26340317c0 con 0x7f26400af4d0 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 10 monclient: dump: 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:03.262 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 10 monclient: _finish_auth 0 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 10 monclient: _check_auth_tickets 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 1 -- 172.21.15.69:0/3579831964 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2634031920 con 0x7f26400af4d0 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f2645cc9640 5 monclient: authenticate success, global_id 17795 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f2645cc9640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 1 -- 172.21.15.69:0/3579831964 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2634031cb0 con 0x7f26400af4d0 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f2645cc9640 4 set_mon_vals no callback set 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.261+0000 7f263e7fc640 10 monclient: dump: 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:03.263 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.264 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.264 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.264 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:03.264 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.264 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:03.264 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:03.264 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:03.264 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.262+0000 7f2645cc9640 10 monclient: shutdown 2023-12-15T17:24:03.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.262+0000 7f2645cc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:03.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.262+0000 7f2645cc9640 1 -- 172.21.15.69:0/3579831964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400af4d0 msgr2=0x7f2640142040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.262+0000 7f2645cc9640 1 --2- 172.21.15.69:0/3579831964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400af4d0 0x7f2640142040 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 crypto rx=0x7f2634009870 tx=0x7f263402b860 comp rx=0 tx=0).stop 2023-12-15T17:24:03.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.262+0000 7f2645cc9640 1 -- 172.21.15.69:0/3579831964 shutdown_connections 2023-12-15T17:24:03.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.262+0000 7f2645cc9640 1 --2- 172.21.15.69:0/3579831964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400af4d0 0x7f2640142040 unknown :-1 s=CLOSED pgs=3130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.262+0000 7f2645cc9640 1 --2- 172.21.15.69:0/3579831964 >> v2:172.21.15.192:3300/0 conn(0x7f26400af0f0 0x7f26400ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.262+0000 7f2645cc9640 1 -- 172.21.15.69:0/3579831964 >> 172.21.15.69:0/3579831964 conn(0x7f26400b01e0 msgr2=0x7f26400b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:03.267 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.262+0000 7f2645cc9640 1 -- 172.21.15.69:0/3579831964 shutdown_connections 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.262+0000 7f2645cc9640 1 -- 172.21.15.69:0/3579831964 wait complete. 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.263+0000 7f2645cc9640 10 monclient: build_initial_monmap 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.263+0000 7f2645cc9640 10 monclient: monmap: 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.268 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.263+0000 7f2645cc9640 1 librados: starting msgr at 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.263+0000 7f2645cc9640 1 librados: starting objecter 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.263+0000 7f2645cc9640 1 Processor -- start 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.263+0000 7f2645cc9640 1 -- start start 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.263+0000 7f2645cc9640 1 librados: setting wanted keys 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.263+0000 7f2645cc9640 1 librados: calling monclient init 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.263+0000 7f2645cc9640 10 monclient: init 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400adaa0 0x7f264006e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f26400adaa0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26400af0f0 0x7f264013b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 10 monclient(hunting): picked mon.noname-a con 0x7f26400af0f0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26400acfc0 0x7f264013b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 10 monclient(hunting): picked mon.noname-b con 0x7f26400acfc0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.269 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26401369a0 con 0x7f26400af0f0 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264013bca0 con 0x7f26400acfc0 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2640136480 con 0x7f26400adaa0 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f2645cc9640 10 monclient(hunting): authenticate will time out at 2747.579730s 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f263f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400adaa0 0x7f264006e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f263effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26400af0f0 0x7f264013b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.264+0000 7f263ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26400acfc0 0x7f264013b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400adaa0 0x7f264006e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42372/0 (socket says 172.21.15.69:42372) 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263f7fe640 1 -- 172.21.15.69:0/1002559561 learned_addr learned my addr 172.21.15.69:0/1002559561 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263f7fe640 10 monclient(hunting): get_auth_request con 0x7f26400adaa0 auth_method 0 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.270 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263ffff640 10 monclient(hunting): get_auth_request con 0x7f26400acfc0 auth_method 0 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263ffff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263effd640 10 monclient(hunting): get_auth_request con 0x7f26400af0f0 auth_method 0 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263effd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.271 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.265+0000 7f263effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.268+0000 7f263ffff640 10 monclient(hunting): handle_auth_done global_id 21454 payload 824 2023-12-15T17:24:03.272 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.268+0000 7f263ffff640 1 -- 172.21.15.69:0/1002559561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400adaa0 msgr2=0x7f264006e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.268+0000 7f263ffff640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400adaa0 0x7f264006e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.268+0000 7f263ffff640 1 -- 172.21.15.69:0/1002559561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26400af0f0 msgr2=0x7f264013b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.268+0000 7f263ffff640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26400af0f0 0x7f264013b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.268+0000 7f263ffff640 10 monclient: _finish_hunting 0 2023-12-15T17:24:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.268+0000 7f263ffff640 1 monclient: found mon.noname-b 2023-12-15T17:24:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.268+0000 7f263ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.268+0000 7f263ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.268+0000 7f263ffff640 1 -- 172.21.15.69:0/1002559561 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26340094d0 con 0x7f26400acfc0 2023-12-15T17:24:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.270+0000 7f263ffff640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26400acfc0 0x7f264013b760 secure :-1 s=READY pgs=5109 cs=0 l=1 rev1=1 crypto rx=0x7f263800c690 tx=0x7f263800cb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.270+0000 7f2627fff640 10 client.?.objecter ms_handle_connect 0x7f26400acfc0 2023-12-15T17:24:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.270+0000 7f2627fff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:03.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.273+0000 7f2627fff640 1 -- 172.21.15.69:0/1002559561 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638003b70 con 0x7f26400acfc0 2023-12-15T17:24:03.274 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.273+0000 7f2627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.273+0000 7f2627fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.273+0000 7f2627fff640 10 monclient: dump: 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.275 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2627fff640 10 monclient: _finish_auth 0 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2627fff640 10 monclient: _check_auth_tickets 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2627fff640 1 -- 172.21.15.69:0/1002559561 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2638003cd0 con 0x7f26400acfc0 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2645cc9640 5 monclient: authenticate success, global_id 21454 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2645cc9640 10 monclient: _renew_subs 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2645cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2645cc9640 1 -- 172.21.15.69:0/1002559561 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2640135b70 con 0x7f26400acfc0 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2627fff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2627fff640 1 -- 172.21.15.69:0/1002559561 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638005310 con 0x7f26400acfc0 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2627fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2627fff640 10 monclient: dump: 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f263dffb640 4 set_mon_vals no callback set 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2645cc9640 10 client.21454.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2645cc9640 10 monclient: _renew_subs 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2645cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2645cc9640 1 -- 172.21.15.69:0/1002559561 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26400ad680 con 0x7f26400acfc0 2023-12-15T17:24:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.274+0000 7f2645cc9640 1 librados: init done 2023-12-15T17:24:03.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.275+0000 7f2645cc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:03.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.275+0000 7f2645cc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:03.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.275+0000 7f2645cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:03.278 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.275+0000 7f2645cc9640 1 -- 172.21.15.69:0/1002559561 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26400b0fe0 con 0x7f26400acfc0 2023-12-15T17:24:03.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.294+0000 7f2627fff640 1 -- 172.21.15.69:0/1002559561 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2638005470 con 0x7f26400acfc0 2023-12-15T17:24:03.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.294+0000 7f2627fff640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f262c06cb80 0x7f262c06f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.295+0000 7f263f7fe640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f262c06cb80 0x7f262c06f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.295+0000 7f263f7fe640 10 monclient: get_auth_request con 0x7f262c06cb80 auth_method 0 2023-12-15T17:24:03.295 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.295+0000 7f263f7fe640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f262c06cb80 0x7f262c06f040 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2630009c00 tx=0x7f2630009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:03.296 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.295+0000 7f2627fff640 10 client.21454.objecter ms_handle_connect 0x7f262c06cb80 2023-12-15T17:24:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.521+0000 7f2627fff640 1 -- 172.21.15.69:0/1002559561 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 534..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7f2638058bc0 con 0x7f26400acfc0 2023-12-15T17:24:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.521+0000 7f2627fff640 10 client.21454.objecter ms_dispatch 0x7f2640000b90 osd_map(908..908 src has 534..908) v4 2023-12-15T17:24:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.521+0000 7f2627fff640 3 client.21454.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.521+0000 7f2627fff640 3 client.21454.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:03.522 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.521+0000 7f2627fff640 20 client.21454.objecter dump_active .. 0 homeless 2023-12-15T17:24:03.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.531+0000 7f2627fff640 1 -- 172.21.15.69:0/1002559561 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2638054f00 con 0x7f26400acfc0 2023-12-15T17:24:03.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.531+0000 7f2627fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:03.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.531+0000 7f2627fff640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:03.714 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:13.660 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:03.719 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:13.665 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:03.755 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.755+0000 7f2645cc9640 1 -- 172.21.15.69:0/1002559561 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2640072970 con 0x7f262c06cb80 2023-12-15T17:24:03.756 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.755+0000 7f2627fff640 1 -- 172.21.15.69:0/1002559561 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2630011b20 con 0x7f262c06cb80 2023-12-15T17:24:03.756 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 1 to deep-scrub 2023-12-15T17:24:03.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 10 librados: watch_flush enter 2023-12-15T17:24:03.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 10 librados: watch_flush exit 2023-12-15T17:24:03.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 20 client.21454.objecter shutdown clearing up homeless session... 2023-12-15T17:24:03.758 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 10 client.21454.objecter successfully canceled tick 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 -- 172.21.15.69:0/1002559561 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f262c06cb80 msgr2=0x7f262c06f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f262c06cb80 0x7f262c06f040 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2630009c00 tx=0x7f2630009190 comp rx=0 tx=0).stop 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 10 monclient: shutdown 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 -- 172.21.15.69:0/1002559561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26400acfc0 msgr2=0x7f264013b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26400acfc0 0x7f264013b760 secure :-1 s=READY pgs=5109 cs=0 l=1 rev1=1 crypto rx=0x7f263800c690 tx=0x7f263800cb50 comp rx=0 tx=0).stop 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 -- 172.21.15.69:0/1002559561 shutdown_connections 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f262c06cb80 0x7f262c06f040 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26400acfc0 0x7f264013b760 unknown :-1 s=CLOSED pgs=5109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26400af0f0 0x7f264013b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 --2- 172.21.15.69:0/1002559561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26400adaa0 0x7f264006e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 -- 172.21.15.69:0/1002559561 >> 172.21.15.69:0/1002559561 conn(0x7f26400b01e0 msgr2=0x7f26400b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 -- 172.21.15.69:0/1002559561 shutdown_connections 2023-12-15T17:24:03.759 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 -- 172.21.15.69:0/1002559561 wait complete. 2023-12-15T17:24:03.760 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.757+0000 7f2645cc9640 1 librados: shutdown 2023-12-15T17:24:03.760 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:13.704 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:03.770 INFO:tasks.ceph:Scrubbing osd.2 2023-12-15T17:24:03.770 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-15T17:24:03.773 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:13.713 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:03.775 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:13.715 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:03.779 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:13.725 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:03.782 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:13.728 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:03.797 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:13.737 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:03.808 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:13.753 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:03.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.882+0000 7f5376e7c640 10 monclient: get_monmap_and_config 2023-12-15T17:24:03.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.882+0000 7f5376e7c640 10 monclient: build_initial_monmap 2023-12-15T17:24:03.882 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.882+0000 7f5376e7c640 10 monclient: monmap: 2023-12-15T17:24:03.882 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:03.883 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.883 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:03.882508+0000 2023-12-15T17:24:03.883 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:03.882508+0000 2023-12-15T17:24:03.883 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:03.883 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.883 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:03.883 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:03.883 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:03.883 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:03.883 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:03.883 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.882+0000 7f5376e7c640 1 Processor -- start 2023-12-15T17:24:03.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.882+0000 7f5376e7c640 1 -- start start 2023-12-15T17:24:03.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.882+0000 7f5376e7c640 10 monclient: init 2023-12-15T17:24:03.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.883+0000 7f5376e7c640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:03.884 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.883+0000 7f5376e7c640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-12-15T17:24:03.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.883+0000 7f5376e7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5370145840 0x7f5370146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.883+0000 7f5376e7c640 10 monclient(hunting): picked mon.noname-b con 0x7f5370145840 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:03.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.883+0000 7f5376e7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53700b20f0 addr v1:172.21.15.69:6789/0 2023-12-15T17:24:03.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.884+0000 7f5376e7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53700b24f0 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:03.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.884+0000 7f5376e7c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53700aa850 con 0x7f53700b20f0 2023-12-15T17:24:03.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.884+0000 7f5376e7c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53700a9260 con 0x7f53700b24f0 2023-12-15T17:24:03.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.884+0000 7f5376e7c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.884+0000 7f5376e7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53701465c0 con 0x7f5370145840 2023-12-15T17:24:03.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.884+0000 7f5376e7c640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:03.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.884+0000 7f5376e7c640 10 monclient(hunting): authenticate will time out at 2748.199420s 2023-12-15T17:24:03.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.888+0000 7f536ffff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f53700b20f0 0x7f537013f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42690/0 (socket says 172.21.15.69:42690) 2023-12-15T17:24:03.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.888+0000 7f536ffff640 1 -- 172.21.15.69:0/4175030616 learned_addr learned my addr 172.21.15.69:0/4175030616 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:24:03.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.900+0000 7f5374c18640 1 --2- 172.21.15.69:0/4175030616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5370145840 0x7f5370146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.907 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.907+0000 7f5374c18640 10 monclient(hunting): get_auth_request con 0x7f5370145840 auth_method 0 2023-12-15T17:24:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.907+0000 7f5374c18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.907+0000 7f5374c18640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.907+0000 7f5374c18640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.907+0000 7f536f7fe640 1 -- 172.21.15.69:0/4175030616 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283773067 0 0) 0x7f53700aa850 con 0x7f53700b20f0 2023-12-15T17:24:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.907+0000 7f536f7fe640 10 monclient(hunting): _negotiate 2023-12-15T17:24:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.907+0000 7f536f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.907+0000 7f536f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.907+0000 7f536f7fe640 10 monclient(hunting): my global_id is 16089 2023-12-15T17:24:03.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.907+0000 7f536f7fe640 1 -- 172.21.15.69:0/4175030616 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5360003520 con 0x7f53700b20f0 2023-12-15T17:24:03.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.908+0000 7f5374c18640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.908+0000 7f5374c18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.908+0000 7f5374c18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.912+0000 7f5374c18640 10 monclient(hunting): handle_auth_done global_id 17801 payload 274 2023-12-15T17:24:03.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.912+0000 7f5374c18640 1 -- 172.21.15.69:0/4175030616 >> v1:172.21.15.192:6789/0 conn(0x7f53700b24f0 legacy=0x7f537014c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:03.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.912+0000 7f5374c18640 1 -- 172.21.15.69:0/4175030616 >> v1:172.21.15.69:6789/0 conn(0x7f53700b20f0 legacy=0x7f537013f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.912+0000 7f5374c18640 10 monclient: _finish_hunting 0 2023-12-15T17:24:03.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.912+0000 7f5374c18640 1 monclient: found mon.noname-b 2023-12-15T17:24:03.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.912+0000 7f5374c18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:03.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.912+0000 7f5374c18640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:24:03.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.912+0000 7f5374c18640 1 -- 172.21.15.69:0/4175030616 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537013fc20 con 0x7f5370145840 2023-12-15T17:24:03.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.926+0000 7f5374c18640 1 --2- 172.21.15.69:0/4175030616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5370145840 0x7f5370146080 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f53640097d0 tx=0x7f536402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42813547d79e47fb server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:03.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 1 -- 172.21.15.69:0/4175030616 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364031770 con 0x7f5370145840 2023-12-15T17:24:03.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:03.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:03.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 10 monclient: dump: 2023-12-15T17:24:03.954 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:03.954 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.954 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.954 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.954 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:03.954 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 10 monclient: _finish_auth 0 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 10 monclient: _check_auth_tickets 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 1 -- 172.21.15.69:0/4175030616 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53640318d0 con 0x7f5370145840 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 1 -- 172.21.15.69:0/4175030616 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364031c60 con 0x7f5370145840 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f5376e7c640 5 monclient: authenticate success, global_id 17801 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f5376e7c640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f5376e7c640 4 set_mon_vals no callback set 2023-12-15T17:24:03.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.953+0000 7f536f7fe640 10 monclient: dump: 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:03.956 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.954+0000 7f5376e7c640 10 monclient: shutdown 2023-12-15T17:24:03.959 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.954+0000 7f5376e7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.954+0000 7f5376e7c640 1 -- 172.21.15.69:0/4175030616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5370145840 msgr2=0x7f5370146080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.954+0000 7f5376e7c640 1 --2- 172.21.15.69:0/4175030616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5370145840 0x7f5370146080 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f53640097d0 tx=0x7f536402b810 comp rx=0 tx=0).stop 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.954+0000 7f5376e7c640 1 -- 172.21.15.69:0/4175030616 shutdown_connections 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.954+0000 7f5376e7c640 1 --2- 172.21.15.69:0/4175030616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5370145840 0x7f5370146080 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.954+0000 7f5376e7c640 1 -- 172.21.15.69:0/4175030616 >> 172.21.15.69:0/4175030616 conn(0x7f53700b41f0 msgr2=0x7f53700b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.954+0000 7f5376e7c640 1 -- 172.21.15.69:0/4175030616 shutdown_connections 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.954+0000 7f5376e7c640 1 -- 172.21.15.69:0/4175030616 wait complete. 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.955+0000 7f5376e7c640 10 monclient: build_initial_monmap 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.955+0000 7f5376e7c640 10 monclient: monmap: 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:03.960 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.955+0000 7f5376e7c640 1 librados: starting msgr at 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.955+0000 7f5376e7c640 1 librados: starting objecter 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.955+0000 7f5376e7c640 1 Processor -- start 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.955+0000 7f5376e7c640 1 -- start start 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.955+0000 7f5376e7c640 1 librados: setting wanted keys 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.955+0000 7f5376e7c640 1 librados: calling monclient init 2023-12-15T17:24:03.961 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.955+0000 7f5376e7c640 10 monclient: init 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53700b20f0 0x7f537013b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 10 monclient(hunting): picked mon.noname-c con 0x7f53700b20f0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53700b24f0 0x7f537013b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 10 monclient(hunting): picked mon.noname-a con 0x7f53700b24f0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5370145840 0x7f53701db110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 10 monclient(hunting): picked mon.noname-b con 0x7f5370145840 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53700af7a0 con 0x7f53700b24f0 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537013bca0 con 0x7f5370145840 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537013c380 con 0x7f53700b20f0 2023-12-15T17:24:03.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:03.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.956+0000 7f5376e7c640 10 monclient(hunting): authenticate will time out at 2748.271989s 2023-12-15T17:24:03.964 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.963+0000 7f536ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53700b24f0 0x7f537013b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.966 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.966+0000 7f5374c18640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53700b20f0 0x7f537013b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.968+0000 7f5374c18640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53700b20f0 0x7f537013b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42382/0 (socket says 172.21.15.69:42382) 2023-12-15T17:24:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.968+0000 7f5374c18640 1 -- 172.21.15.69:0/3623068827 learned_addr learned my addr 172.21.15.69:0/3623068827 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.968+0000 7f5374c18640 10 monclient(hunting): get_auth_request con 0x7f53700b20f0 auth_method 0 2023-12-15T17:24:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.968+0000 7f5374c18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.968+0000 7f5374c18640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.969 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.968+0000 7f5374c18640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.969+0000 7f5374c18640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.969+0000 7f5374c18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.969+0000 7f5374c18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.970+0000 7f536ffff640 10 monclient(hunting): get_auth_request con 0x7f53700b24f0 auth_method 0 2023-12-15T17:24:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.970+0000 7f536ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.970+0000 7f536ffff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.970+0000 7f536ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.971+0000 7f536ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:03.972 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.971+0000 7f536ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:03.972 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.971+0000 7f536ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:03.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.973+0000 7f5374c18640 10 monclient(hunting): handle_auth_done global_id 17804 payload 824 2023-12-15T17:24:03.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.973+0000 7f5374c18640 1 -- 172.21.15.69:0/3623068827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5370145840 msgr2=0x7f53701db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.973+0000 7f5374c18640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5370145840 0x7f53701db110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.973+0000 7f5374c18640 1 -- 172.21.15.69:0/3623068827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53700b24f0 msgr2=0x7f537013b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:03.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.973+0000 7f5374c18640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53700b24f0 0x7f537013b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:03.974 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.973+0000 7f5374c18640 10 monclient: _finish_hunting 0 2023-12-15T17:24:03.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.973+0000 7f5374c18640 1 monclient: found mon.noname-c 2023-12-15T17:24:03.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.973+0000 7f5374c18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:03.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.973+0000 7f5374c18640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:24:03.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.973+0000 7f5374c18640 1 -- 172.21.15.69:0/3623068827 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5364009400 con 0x7f53700b20f0 2023-12-15T17:24:03.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.984+0000 7f5374c18640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53700b20f0 0x7f537013b220 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7f53640097d0 tx=0x7f53640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:03.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.984+0000 7f536d7fa640 10 client.?.objecter ms_handle_connect 0x7f53700b20f0 2023-12-15T17:24:03.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:03.984+0000 7f536d7fa640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:04.062 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 1 -- 172.21.15.69:0/3623068827 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364032810 con 0x7f53700b20f0 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 10 monclient: dump: 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 10 monclient: _finish_auth 0 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 10 monclient: _check_auth_tickets 2023-12-15T17:24:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f5376e7c640 5 monclient: authenticate success, global_id 17804 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 1 -- 172.21.15.69:0/3623068827 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5364032990 con 0x7f53700b20f0 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f5376e7c640 10 monclient: _renew_subs 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f5376e7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f5376e7c640 1 -- 172.21.15.69:0/3623068827 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f537013c550 con 0x7f53700b20f0 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 1 -- 172.21.15.69:0/3623068827 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364032d40 con 0x7f53700b20f0 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536d7fa640 10 monclient: dump: 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.064 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f536effd640 4 set_mon_vals no callback set 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f5376e7c640 10 client.17804.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f5376e7c640 10 monclient: _renew_subs 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f5376e7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f5376e7c640 1 -- 172.21.15.69:0/3623068827 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53701db830 con 0x7f53700b20f0 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.061+0000 7f5376e7c640 1 librados: init done 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.062+0000 7f5376e7c640 10 client.17804.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.062+0000 7f5376e7c640 15 client.17804.objecter _session_command_op_assign -1 1 2023-12-15T17:24:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.062+0000 7f5376e7c640 20 client.17804.objecter _get_session osd=-1 returning homeless 2023-12-15T17:24:04.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.062+0000 7f5376e7c640 10 client.17804.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:04.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.062+0000 7f5376e7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:04.066 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.062+0000 7f5376e7c640 1 -- 172.21.15.69:0/3623068827 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f53700aad80 con 0x7f53700b20f0 2023-12-15T17:24:04.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.106+0000 7f536d7fa640 1 -- 172.21.15.69:0/3623068827 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f536404d3f0 con 0x7f53700b20f0 2023-12-15T17:24:04.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.106+0000 7f536d7fa640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f535406cba0 0x7f535406f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:04.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.107+0000 7f536ffff640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f535406cba0 0x7f535406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:04.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.107+0000 7f536ffff640 10 monclient: get_auth_request con 0x7f535406cba0 auth_method 0 2023-12-15T17:24:04.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.107+0000 7f536d7fa640 10 client.17804.objecter ms_handle_connect 0x7f535406cba0 2023-12-15T17:24:04.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.107+0000 7f536ffff640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f535406cba0 0x7f535406f060 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f53700b1770 tx=0x7f535c007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:04.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 1 -- 172.21.15.69:0/3623068827 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7f536407a700 con 0x7f53700b20f0 2023-12-15T17:24:04.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 10 client.17804.objecter ms_dispatch 0x7f5370000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:04.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 3 client.17804.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 3 client.17804.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 10 client.17804.objecter checking command 1 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x7f5354074430 0x7f53540768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 20 client.17804.objecter _get_session s=0x7f5354073d60 osd=2 3 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 20 client.17804.objecter put_session s=0x7f5354073d60 osd=2 3 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 15 client.17804.objecter _session_command_op_remove -1 1 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 20 client.17804.objecter _get_session s=0x7f5354073d60 osd=2 3 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 20 client.17804.objecter get_session s=0x7f5354073d60 osd=2 3 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 15 client.17804.objecter _session_command_op_assign 2 1 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 20 client.17804.objecter put_session s=0x7f5354073d60 osd=2 4 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 10 client.17804.objecter _send_command 1 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 1 -- 172.21.15.69:0/3623068827 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5354076fa0 con 0x7f5354074430 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 20 client.17804.objecter dump_active .. 0 homeless 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 1 -- 172.21.15.69:0/3623068827 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=908) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5364032ea0 con 0x7f53700b20f0 2023-12-15T17:24:04.112 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f536d7fa640 10 monclient: handle_get_version_reply finishing 1 version 908 2023-12-15T17:24:04.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.111+0000 7f5375419640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x7f5354074430 0x7f53540768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:04.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.112+0000 7f5375419640 10 monclient: get_auth_request con 0x7f5354074430 auth_method 0 2023-12-15T17:24:04.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.114+0000 7f536d7fa640 10 client.17804.objecter ms_handle_connect 0x7f5354074430 2023-12-15T17:24:04.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.114+0000 7f5375419640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x7f5354074430 0x7f53540768d0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:04.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.145+0000 7f536d7fa640 1 -- 172.21.15.69:0/3623068827 <== osd.2 v2:172.21.15.69:6800/3322029776 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5354076fa0 con 0x7f5354074430 2023-12-15T17:24:04.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.145+0000 7f536d7fa640 10 client.17804.objecter ms_dispatch 0x7f5370000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:24:04.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.145+0000 7f536d7fa640 10 client.17804.objecter _finish_command 1 = system:0 2023-12-15T17:24:04.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.145+0000 7f536d7fa640 20 client.17804.objecter put_session s=0x7f5354073d60 osd=2 4 2023-12-15T17:24:04.146 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.145+0000 7f536d7fa640 15 client.17804.objecter _session_command_op_remove 2 1 2023-12-15T17:24:04.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.169+0000 7f5376e7c640 10 client.17804.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-15T17:24:04.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.169+0000 7f5376e7c640 15 client.17804.objecter _session_command_op_assign -1 2 2023-12-15T17:24:04.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.169+0000 7f5376e7c640 20 client.17804.objecter _get_session s=0x7f5354073d60 osd=2 3 2023-12-15T17:24:04.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.169+0000 7f5376e7c640 20 client.17804.objecter put_session s=0x7f5354073d60 osd=2 3 2023-12-15T17:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.169+0000 7f5376e7c640 20 client.17804.objecter _get_session s=0x7f5354073d60 osd=2 3 2023-12-15T17:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.169+0000 7f5376e7c640 15 client.17804.objecter _session_command_op_remove -1 2 2023-12-15T17:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.169+0000 7f5376e7c640 20 client.17804.objecter get_session s=0x7f5354073d60 osd=2 3 2023-12-15T17:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.169+0000 7f5376e7c640 15 client.17804.objecter _session_command_op_assign 2 2 2023-12-15T17:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.169+0000 7f5376e7c640 20 client.17804.objecter put_session s=0x7f5354073d60 osd=2 4 2023-12-15T17:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.169+0000 7f5376e7c640 10 client.17804.objecter _send_command 2 2023-12-15T17:24:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.169+0000 7f5376e7c640 1 -- 172.21.15.69:0/3623068827 --> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5370145c20 con 0x7f5354074430 2023-12-15T17:24:04.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.191+0000 7f536d7fa640 1 -- 172.21.15.69:0/3623068827 <== osd.2 v2:172.21.15.69:6800/3322029776 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5368006b10 con 0x7f5354074430 2023-12-15T17:24:04.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.191+0000 7f536d7fa640 10 client.17804.objecter ms_dispatch 0x7f5370000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:24:04.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.191+0000 7f536d7fa640 10 client.17804.objecter _finish_command 2 = system:0 2023-12-15T17:24:04.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.191+0000 7f536d7fa640 20 client.17804.objecter put_session s=0x7f5354073d60 osd=2 4 2023-12-15T17:24:04.192 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.191+0000 7f536d7fa640 15 client.17804.objecter _session_command_op_remove 2 2 2023-12-15T17:24:04.192 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-12-15T17:24:04.192 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-15T17:24:04.193 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 10 librados: watch_flush enter 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 10 librados: watch_flush exit 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 10 client.17804.objecter close_session for osd.2 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 1 -- 172.21.15.69:0/3623068827 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x7f5354074430 msgr2=0x7f53540768d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.69:6800/3322029776,v1:172.21.15.69:6801/3322029776] conn(0x7f5354074430 0x7f53540768d0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 20 client.17804.objecter put_session s=0x7f5354073d60 osd=2 1 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 20 client.17804.objecter shutdown clearing up homeless session... 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 10 client.17804.objecter successfully canceled tick 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 1 -- 172.21.15.69:0/3623068827 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f535406cba0 msgr2=0x7f535406f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f535406cba0 0x7f535406f060 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f53700b1770 tx=0x7f535c007d30 comp rx=0 tx=0).stop 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 10 monclient: shutdown 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 1 -- 172.21.15.69:0/3623068827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53700b20f0 msgr2=0x7f537013b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:04.194 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 1 --2- 172.21.15.69:0/3623068827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53700b20f0 0x7f537013b220 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7f53640097d0 tx=0x7f53640046c0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5375419640 1 -- 172.21.15.69:0/3623068827 reap_dead start 2023-12-15T17:24:04.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 1 -- 172.21.15.69:0/3623068827 shutdown_connections 2023-12-15T17:24:04.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 1 -- 172.21.15.69:0/3623068827 >> 172.21.15.69:0/3623068827 conn(0x7f53700b41f0 msgr2=0x7f53700b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:04.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 1 -- 172.21.15.69:0/3623068827 shutdown_connections 2023-12-15T17:24:04.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 1 -- 172.21.15.69:0/3623068827 wait complete. 2023-12-15T17:24:04.195 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.192+0000 7f5376e7c640 1 librados: shutdown 2023-12-15T17:24:04.204 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-15T17:24:04.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.346+0000 7faa91494640 10 monclient: get_monmap_and_config 2023-12-15T17:24:04.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.346+0000 7faa91494640 10 monclient: build_initial_monmap 2023-12-15T17:24:04.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.347+0000 7faa91494640 10 monclient: monmap: 2023-12-15T17:24:04.347 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:04.347 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:04.347 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:04.347461+0000 2023-12-15T17:24:04.348 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:04.347461+0000 2023-12-15T17:24:04.348 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:04.348 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:04.348 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:04.348 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:04.348 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:04.348 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:04.348 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:04.348 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.347+0000 7faa91494640 1 Processor -- start 2023-12-15T17:24:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.347+0000 7faa91494640 1 -- start start 2023-12-15T17:24:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.347+0000 7faa91494640 10 monclient: init 2023-12-15T17:24:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa8c0a8910 addr v1:172.21.15.69:6789/0 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faa8c0a9b70 0x7faa8c13f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 10 monclient(hunting): picked mon.noname-a con 0x7faa8c0a9b70 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 0x7faa8c1423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 10 monclient(hunting): picked mon.noname-b con 0x7faa8c141f90 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0a8cf0 con 0x7faa8c0a8910 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c0aabd0 con 0x7faa8c0a9b70 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0aaeb0 con 0x7faa8c141f90 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:04.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.348+0000 7faa91494640 10 monclient(hunting): authenticate will time out at 2748.663608s 2023-12-15T17:24:04.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.350+0000 7faa8b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 0x7faa8c1423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:04.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.350+0000 7faa8a7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faa8c0a9b70 0x7faa8c13f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:04.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.352+0000 7faa8b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 0x7faa8c1423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42392/0 (socket says 172.21.15.69:42392) 2023-12-15T17:24:04.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.352+0000 7faa8b7fe640 1 -- 172.21.15.69:0/1953159654 learned_addr learned my addr 172.21.15.69:0/1953159654 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:04.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.352+0000 7faa8b7fe640 10 monclient(hunting): get_auth_request con 0x7faa8c141f90 auth_method 0 2023-12-15T17:24:04.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.352+0000 7faa8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:04.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.352+0000 7faa8b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:04.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.352+0000 7faa8b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:04.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.352+0000 7faa8a7fc640 10 monclient(hunting): get_auth_request con 0x7faa8c0a9b70 auth_method 0 2023-12-15T17:24:04.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.352+0000 7faa8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:04.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.352+0000 7faa8a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:04.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.352+0000 7faa8a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:04.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.353+0000 7faa8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:04.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.353+0000 7faa8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:04.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.353+0000 7faa8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:04.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.353+0000 7faa8a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:04.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.353+0000 7faa8a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:04.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.353+0000 7faa8a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:04.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.356+0000 7faa8b7fe640 10 monclient(hunting): handle_auth_done global_id 17807 payload 274 2023-12-15T17:24:04.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.356+0000 7faa8b7fe640 1 -- 172.21.15.69:0/1953159654 >> v2:172.21.15.69:3300/0 conn(0x7faa8c0a9b70 msgr2=0x7faa8c13f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:04.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.356+0000 7faa8b7fe640 1 --2- 172.21.15.69:0/1953159654 >> v2:172.21.15.69:3300/0 conn(0x7faa8c0a9b70 0x7faa8c13f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.356+0000 7faa8b7fe640 1 -- 172.21.15.69:0/1953159654 >> v1:172.21.15.69:6789/0 conn(0x7faa8c0a8910 legacy=0x7faa8c13ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:04.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.356+0000 7faa8b7fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:04.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.356+0000 7faa8b7fe640 1 monclient: found mon.noname-b 2023-12-15T17:24:04.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.356+0000 7faa8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:04.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.356+0000 7faa8b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:24:04.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.356+0000 7faa8b7fe640 1 -- 172.21.15.69:0/1953159654 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c0a9720 con 0x7faa8c141f90 2023-12-15T17:24:04.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.360+0000 7faa8b7fe640 1 --2- 172.21.15.69:0/1953159654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 0x7faa8c1423f0 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7faa8c074590 tx=0x7faa8402beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52ef1738d2003d25 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:04.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 1 -- 172.21.15.69:0/1953159654 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8402ca50 con 0x7faa8c141f90 2023-12-15T17:24:04.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:04.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:04.374 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 10 monclient: dump: 2023-12-15T17:24:04.374 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:04.374 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 10 monclient: _finish_auth 0 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 10 monclient: _check_auth_tickets 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:04.375 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 1 -- 172.21.15.69:0/1953159654 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8402cbb0 con 0x7faa8c141f90 2023-12-15T17:24:04.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa91494640 5 monclient: authenticate success, global_id 17807 2023-12-15T17:24:04.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa91494640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:24:04.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:04.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa91494640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:04.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 1 -- 172.21.15.69:0/1953159654 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa840343f0 con 0x7faa8c141f90 2023-12-15T17:24:04.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa91494640 4 set_mon_vals no callback set 2023-12-15T17:24:04.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:04.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:04.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa89ffb640 10 monclient: dump: 2023-12-15T17:24:04.377 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa91494640 10 monclient: shutdown 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa91494640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:04.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa91494640 1 -- 172.21.15.69:0/1953159654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 msgr2=0x7faa8c1423f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:04.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.373+0000 7faa91494640 1 --2- 172.21.15.69:0/1953159654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 0x7faa8c1423f0 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7faa8c074590 tx=0x7faa8402beb0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.374+0000 7faa91494640 1 -- 172.21.15.69:0/1953159654 shutdown_connections 2023-12-15T17:24:04.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.374+0000 7faa91494640 1 --2- 172.21.15.69:0/1953159654 >> v2:172.21.15.69:3300/0 conn(0x7faa8c0a9b70 0x7faa8c13f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.374+0000 7faa91494640 1 --2- 172.21.15.69:0/1953159654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 0x7faa8c1423f0 unknown :-1 s=CLOSED pgs=3133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.374+0000 7faa91494640 1 -- 172.21.15.69:0/1953159654 >> 172.21.15.69:0/1953159654 conn(0x7faa8c0b7780 msgr2=0x7faa8c0b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:04.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.374+0000 7faa91494640 1 -- 172.21.15.69:0/1953159654 shutdown_connections 2023-12-15T17:24:04.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.374+0000 7faa91494640 1 -- 172.21.15.69:0/1953159654 wait complete. 2023-12-15T17:24:04.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.374+0000 7faa91494640 10 monclient: build_initial_monmap 2023-12-15T17:24:04.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.374+0000 7faa91494640 10 monclient: monmap: 2023-12-15T17:24:04.381 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:04.381 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:04.381 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 1 librados: starting msgr at 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 1 librados: starting objecter 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 1 Processor -- start 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 1 -- start start 2023-12-15T17:24:04.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 1 librados: setting wanted keys 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 1 librados: calling monclient init 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 10 monclient: init 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa8c0a8910 0x7faa8c138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 10 monclient(hunting): picked mon.noname-b con 0x7faa8c0a8910 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 0x7faa8c13b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 10 monclient(hunting): picked mon.noname-c con 0x7faa8c141f90 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.375+0000 7faa91494640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8c0ac770 0x7faa8c13bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.376+0000 7faa91494640 10 monclient(hunting): picked mon.noname-a con 0x7faa8c0ac770 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.376+0000 7faa91494640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.376+0000 7faa91494640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c13b130 con 0x7faa8c0ac770 2023-12-15T17:24:04.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.376+0000 7faa91494640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.376+0000 7faa91494640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0abe00 con 0x7faa8c0a8910 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.376+0000 7faa91494640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.376+0000 7faa91494640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0ac070 con 0x7faa8c141f90 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.376+0000 7faa91494640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.376+0000 7faa91494640 10 monclient(hunting): authenticate will time out at 2748.691389s 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.378+0000 7faa8b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8c0ac770 0x7faa8c13bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.380+0000 7faa8b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8c0ac770 0x7faa8c13bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45404/0 (socket says 172.21.15.69:45404) 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.380+0000 7faa8b7fe640 1 -- 172.21.15.69:0/4039380252 learned_addr learned my addr 172.21.15.69:0/4039380252 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.380+0000 7faa8b7fe640 10 monclient(hunting): get_auth_request con 0x7faa8c0ac770 auth_method 0 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.380+0000 7faa8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.380+0000 7faa8b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.380+0000 7faa8b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:04.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.380+0000 7faa8a7fc640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 0x7faa8c13b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:04.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.382+0000 7faa8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:04.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.382+0000 7faa8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:04.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.382+0000 7faa8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:04.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.382+0000 7faa8a7fc640 10 monclient(hunting): get_auth_request con 0x7faa8c141f90 auth_method 0 2023-12-15T17:24:04.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.382+0000 7faa8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:04.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.382+0000 7faa8a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:04.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.382+0000 7faa8a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.383+0000 7faa8a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.383+0000 7faa8a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.383+0000 7faa8a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.383+0000 7faa8affd640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa8c0a8910 0x7faa8c138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8affd640 10 monclient(hunting): get_auth_request con 0x7faa8c0a8910 auth_method 0 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8affd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8b7fe640 10 monclient(hunting): handle_auth_done global_id 16098 payload 824 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8b7fe640 1 -- 172.21.15.69:0/4039380252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 msgr2=0x7faa8c13b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8b7fe640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 0x7faa8c13b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8b7fe640 1 -- 172.21.15.69:0/4039380252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa8c0a8910 msgr2=0x7faa8c138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8b7fe640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa8c0a8910 0x7faa8c138220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8b7fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:04.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8b7fe640 1 monclient: found mon.noname-a 2023-12-15T17:24:04.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:04.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:24:04.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.385+0000 7faa8b7fe640 1 -- 172.21.15.69:0/4039380252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa84009b30 con 0x7faa8c0ac770 2023-12-15T17:24:04.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.390+0000 7faa8b7fe640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8c0ac770 0x7faa8c13bbb0 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7faa84009f40 tx=0x7faa84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:04.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.390+0000 7faa737fe640 10 client.?.objecter ms_handle_connect 0x7faa8c0ac770 2023-12-15T17:24:04.391 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.390+0000 7faa737fe640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:04.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.404+0000 7faa737fe640 1 -- 172.21.15.69:0/4039380252 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84040670 con 0x7faa8c0ac770 2023-12-15T17:24:04.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:04.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:24:04.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 10 monclient: dump: 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 10 monclient: _finish_auth 0 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 10 monclient: _check_auth_tickets 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 1 -- 172.21.15.69:0/4039380252 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa840407d0 con 0x7faa8c0ac770 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:04.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 1 -- 172.21.15.69:0/4039380252 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84040b80 con 0x7faa8c0ac770 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa737fe640 10 monclient: dump: 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa91494640 5 monclient: authenticate success, global_id 16098 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa91494640 10 monclient: _renew_subs 2023-12-15T17:24:04.407 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa91494640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:24:04.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa91494640 1 -- 172.21.15.69:0/4039380252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c139fd0 con 0x7faa8c0ac770 2023-12-15T17:24:04.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa897fa640 4 set_mon_vals no callback set 2023-12-15T17:24:04.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa91494640 10 client.16098.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:04.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa91494640 10 monclient: _renew_subs 2023-12-15T17:24:04.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa91494640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:24:04.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa91494640 1 -- 172.21.15.69:0/4039380252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c13ad60 con 0x7faa8c0ac770 2023-12-15T17:24:04.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.405+0000 7faa91494640 1 librados: init done 2023-12-15T17:24:04.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.406+0000 7faa91494640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:04.408 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.406+0000 7faa91494640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:04.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.406+0000 7faa91494640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:24:04.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.406+0000 7faa91494640 1 -- 172.21.15.69:0/4039380252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c13af20 con 0x7faa8c0ac770 2023-12-15T17:24:04.426 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:14.365 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:04.427 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:14.366 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:04.434 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:14.374 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:04.602 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:14.548 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:04.603 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:14.549 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:04.611 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:14.557 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.671+0000 7faa737fe640 1 -- 172.21.15.69:0/4039380252 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7faa84040ce0 con 0x7faa8c0ac770 2023-12-15T17:24:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.672+0000 7faa737fe640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7faa7806cba0 0x7faa7806f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.672+0000 7faa8affd640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7faa7806cba0 0x7faa7806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.672+0000 7faa8affd640 10 monclient: get_auth_request con 0x7faa7806cba0 auth_method 0 2023-12-15T17:24:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.672+0000 7faa737fe640 10 client.16098.objecter ms_handle_connect 0x7faa7806cba0 2023-12-15T17:24:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.672+0000 7faa8affd640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7faa7806cba0 0x7faa7806f060 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faa7c007fd0 tx=0x7faa7c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:04.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.687+0000 7faa737fe640 1 -- 172.21.15.69:0/4039380252 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7faa8407c370 con 0x7faa8c0ac770 2023-12-15T17:24:04.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.687+0000 7faa737fe640 10 client.16098.objecter ms_dispatch 0x7faa8c000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:04.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.687+0000 7faa737fe640 3 client.16098.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:04.688 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.687+0000 7faa737fe640 3 client.16098.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:04.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.688+0000 7faa737fe640 20 client.16098.objecter dump_active .. 0 homeless 2023-12-15T17:24:04.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.688+0000 7faa737fe640 1 -- 172.21.15.69:0/4039380252 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7faa84077fb0 con 0x7faa8c0ac770 2023-12-15T17:24:04.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.688+0000 7faa737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:04.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.688+0000 7faa737fe640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:04.828 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:14.774 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:04.830 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:14.776 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:04.840 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:14.786 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:04.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.895+0000 7faa91494640 1 -- 172.21.15.69:0/4039380252 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7faa8c13b3d0 con 0x7faa7806cba0 2023-12-15T17:24:04.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.895+0000 7faa737fe640 1 -- 172.21.15.69:0/4039380252 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7faa8c13b3d0 con 0x7faa7806cba0 2023-12-15T17:24:04.896 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 2 to deep-scrub 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 10 librados: watch_flush enter 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 10 librados: watch_flush exit 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 20 client.16098.objecter shutdown clearing up homeless session... 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 10 client.16098.objecter successfully canceled tick 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 -- 172.21.15.69:0/4039380252 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7faa7806cba0 msgr2=0x7faa7806f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7faa7806cba0 0x7faa7806f060 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faa7c007fd0 tx=0x7faa7c009040 comp rx=0 tx=0).stop 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 10 monclient: shutdown 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 -- 172.21.15.69:0/4039380252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8c0ac770 msgr2=0x7faa8c13bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8c0ac770 0x7faa8c13bbb0 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7faa84009f40 tx=0x7faa84004a40 comp rx=0 tx=0).stop 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 -- 172.21.15.69:0/4039380252 shutdown_connections 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7faa7806cba0 0x7faa7806f060 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa8c0ac770 0x7faa8c13bbb0 unknown :-1 s=CLOSED pgs=3162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8c141f90 0x7faa8c13b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 --2- 172.21.15.69:0/4039380252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa8c0a8910 0x7faa8c138220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:04.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 -- 172.21.15.69:0/4039380252 >> 172.21.15.69:0/4039380252 conn(0x7faa8c0b7780 msgr2=0x7faa8c134520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:04.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 -- 172.21.15.69:0/4039380252 shutdown_connections 2023-12-15T17:24:04.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 -- 172.21.15.69:0/4039380252 wait complete. 2023-12-15T17:24:04.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:04.898+0000 7faa91494640 1 librados: shutdown 2023-12-15T17:24:04.912 INFO:tasks.ceph:Scrubbing osd.3 2023-12-15T17:24:04.912 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.021+0000 7fcb71da9640 10 monclient: get_monmap_and_config 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.021+0000 7fcb71da9640 10 monclient: build_initial_monmap 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.021+0000 7fcb71da9640 10 monclient: monmap: 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:05.022169+0000 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:05.022169+0000 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:05.022 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:05.023 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.022+0000 7fcb71da9640 1 Processor -- start 2023-12-15T17:24:05.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.022+0000 7fcb71da9640 1 -- start start 2023-12-15T17:24:05.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.022+0000 7fcb71da9640 10 monclient: init 2023-12-15T17:24:05.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:05.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-12-15T17:24:05.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcb6c145840 0x7fcb6c146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6c145840 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:05.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb6c0b20f0 addr v1:172.21.15.69:6789/0 2023-12-15T17:24:05.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb6c0b2520 0x7fcb6c14c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.024 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 10 monclient(hunting): picked mon.noname-b con 0x7fcb6c0b2520 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c0aa850 con 0x7fcb6c0b20f0 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6c1465c0 con 0x7fcb6c145840 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c145c20 con 0x7fcb6c0b2520 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb71da9640 10 monclient(hunting): authenticate will time out at 2749.338557s 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.023+0000 7fcb6b7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcb6c145840 0x7fcb6c146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.024+0000 7fcb6b7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcb6c145840 0x7fcb6c146080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:43952/0 (socket says 172.21.15.69:43952) 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.024+0000 7fcb6b7fe640 1 -- 172.21.15.69:0/3143204065 learned_addr learned my addr 172.21.15.69:0/3143204065 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.024+0000 7fcb6b7fe640 10 monclient(hunting): get_auth_request con 0x7fcb6c145840 auth_method 0 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.024+0000 7fcb6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.024+0000 7fcb6b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.025 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.024+0000 7fcb6b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.024+0000 7fcb6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:05.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.024+0000 7fcb6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:05.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.024+0000 7fcb6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:05.026 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.025+0000 7fcb6bfff640 1 --2- 172.21.15.69:0/3143204065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb6c0b2520 0x7fcb6c14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.026+0000 7fcb6b7fe640 10 monclient(hunting): handle_auth_done global_id 21460 payload 274 2023-12-15T17:24:05.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.026+0000 7fcb6b7fe640 1 -- 172.21.15.69:0/3143204065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb6c0b2520 msgr2=0x7fcb6c14c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:05.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.026+0000 7fcb6b7fe640 1 --2- 172.21.15.69:0/3143204065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb6c0b2520 0x7fcb6c14c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.026+0000 7fcb6b7fe640 1 -- 172.21.15.69:0/3143204065 >> v1:172.21.15.69:6789/0 conn(0x7fcb6c0b20f0 legacy=0x7fcb6c13f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:05.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.026+0000 7fcb6b7fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:05.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.026+0000 7fcb6b7fe640 1 monclient: found mon.noname-c 2023-12-15T17:24:05.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.026+0000 7fcb6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:05.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.026+0000 7fcb6b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.026+0000 7fcb6b7fe640 1 -- 172.21.15.69:0/3143204065 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c0a9260 con 0x7fcb6c145840 2023-12-15T17:24:05.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.028+0000 7fcb6b7fe640 1 --2- 172.21.15.69:0/3143204065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb6c145840 0x7fcb6c146080 secure :-1 s=READY pgs=5110 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009e80 tx=0x7fcb5c02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea92bb4fe447d7e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:05.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 1 -- 172.21.15.69:0/3143204065 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c02c9c0 con 0x7fcb6c145840 2023-12-15T17:24:05.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:05.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 10 monclient: dump: 2023-12-15T17:24:05.035 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.035 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.035 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.035 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.035 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.035 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.035 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 10 monclient: _finish_auth 0 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 10 monclient: _check_auth_tickets 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb71da9640 5 monclient: authenticate success, global_id 21460 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 1 -- 172.21.15.69:0/3143204065 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c02cb20 con 0x7fcb6c145840 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb71da9640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 1 -- 172.21.15.69:0/3143204065 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0343f0 con 0x7fcb6c145840 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb71da9640 4 set_mon_vals no callback set 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:05.036 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.031+0000 7fcb6a7fc640 10 monclient: dump: 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.032+0000 7fcb71da9640 10 monclient: shutdown 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.032+0000 7fcb71da9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.032+0000 7fcb71da9640 1 -- 172.21.15.69:0/3143204065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb6c145840 msgr2=0x7fcb6c146080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.032+0000 7fcb71da9640 1 --2- 172.21.15.69:0/3143204065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb6c145840 0x7fcb6c146080 secure :-1 s=READY pgs=5110 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009e80 tx=0x7fcb5c02be20 comp rx=0 tx=0).stop 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.032+0000 7fcb71da9640 1 -- 172.21.15.69:0/3143204065 shutdown_connections 2023-12-15T17:24:05.037 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.032+0000 7fcb71da9640 1 --2- 172.21.15.69:0/3143204065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb6c0b2520 0x7fcb6c14c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.032+0000 7fcb71da9640 1 --2- 172.21.15.69:0/3143204065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb6c145840 0x7fcb6c146080 unknown :-1 s=CLOSED pgs=5110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.032+0000 7fcb71da9640 1 -- 172.21.15.69:0/3143204065 >> 172.21.15.69:0/3143204065 conn(0x7fcb6c0b41f0 msgr2=0x7fcb6c0b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.032+0000 7fcb71da9640 1 -- 172.21.15.69:0/3143204065 shutdown_connections 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.032+0000 7fcb71da9640 1 -- 172.21.15.69:0/3143204065 wait complete. 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.033+0000 7fcb71da9640 10 monclient: build_initial_monmap 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.033+0000 7fcb71da9640 10 monclient: monmap: 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:05.038 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.033+0000 7fcb71da9640 1 librados: starting msgr at 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.033+0000 7fcb71da9640 1 librados: starting objecter 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.033+0000 7fcb71da9640 1 Processor -- start 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.033+0000 7fcb71da9640 1 -- start start 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.033+0000 7fcb71da9640 1 librados: setting wanted keys 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.033+0000 7fcb71da9640 1 librados: calling monclient init 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.033+0000 7fcb71da9640 10 monclient: init 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb6c0b20f0 0x7fcb6c1dac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6c0b20f0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb6c0b2520 0x7fcb6c1cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 10 monclient(hunting): picked mon.noname-b con 0x7fcb6c0b2520 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb6c145840 0x7fcb6c13b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 10 monclient(hunting): picked mon.noname-a con 0x7fcb6c145840 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:05.039 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c1db1b0 con 0x7fcb6c145840 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0ad260 con 0x7fcb6c0b2520 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0ad490 con 0x7fcb6c0b20f0 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb71da9640 10 monclient(hunting): authenticate will time out at 2749.349541s 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.034+0000 7fcb6affd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb6c0b2520 0x7fcb6c1cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.035+0000 7fcb6affd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb6c0b2520 0x7fcb6c1cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:43968/0 (socket says 172.21.15.69:43968) 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.035+0000 7fcb6affd640 1 -- 172.21.15.69:0/4230168939 learned_addr learned my addr 172.21.15.69:0/4230168939 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.035+0000 7fcb6affd640 10 monclient(hunting): get_auth_request con 0x7fcb6c0b2520 auth_method 0 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.035+0000 7fcb6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.035+0000 7fcb6affd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.040 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.035+0000 7fcb6affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.035+0000 7fcb6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.035+0000 7fcb6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.035+0000 7fcb6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.035+0000 7fcb6b7fe640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb6c0b20f0 0x7fcb6c1dac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.037+0000 7fcb6b7fe640 10 monclient(hunting): get_auth_request con 0x7fcb6c0b20f0 auth_method 0 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.037+0000 7fcb6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.037+0000 7fcb6b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.037+0000 7fcb6b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.038+0000 7fcb6affd640 10 monclient(hunting): handle_auth_done global_id 21463 payload 824 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.038+0000 7fcb6affd640 1 -- 172.21.15.69:0/4230168939 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb6c0b20f0 msgr2=0x7fcb6c1dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.038+0000 7fcb6affd640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb6c0b20f0 0x7fcb6c1dac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.038+0000 7fcb6affd640 1 -- 172.21.15.69:0/4230168939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb6c145840 msgr2=0x7fcb6c13b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.038+0000 7fcb6affd640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb6c145840 0x7fcb6c13b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.038+0000 7fcb6affd640 10 monclient: _finish_hunting 0 2023-12-15T17:24:05.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.038+0000 7fcb6affd640 1 monclient: found mon.noname-b 2023-12-15T17:24:05.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.038+0000 7fcb6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:05.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.038+0000 7fcb6affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.038+0000 7fcb6affd640 1 -- 172.21.15.69:0/4230168939 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c009b30 con 0x7fcb6c0b2520 2023-12-15T17:24:05.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.040+0000 7fcb6affd640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb6c0b2520 0x7fcb6c1cc6d0 secure :-1 s=READY pgs=5111 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400c410 tx=0x7fcb5400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:05.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.040+0000 7fcb63fff640 10 client.?.objecter ms_handle_connect 0x7fcb6c0b2520 2023-12-15T17:24:05.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.040+0000 7fcb63fff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:05.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 1 -- 172.21.15.69:0/4230168939 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb540035f0 con 0x7fcb6c0b2520 2023-12-15T17:24:05.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:05.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 10 monclient: dump: 2023-12-15T17:24:05.092 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.092 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.092 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.092 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.092 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.092 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 10 monclient: _finish_auth 0 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 10 monclient: _check_auth_tickets 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 1 -- 172.21.15.69:0/4230168939 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb54003750 con 0x7fcb6c0b2520 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb71da9640 5 monclient: authenticate success, global_id 21463 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb71da9640 10 monclient: _renew_subs 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb71da9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb71da9640 1 -- 172.21.15.69:0/4230168939 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c0ad770 con 0x7fcb6c0b2520 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:05.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 1 -- 172.21.15.69:0/4230168939 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb54003b40 con 0x7fcb6c0b2520 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb63fff640 10 monclient: dump: 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb69ffb640 4 set_mon_vals no callback set 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb71da9640 10 client.21463.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:05.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb71da9640 10 monclient: _renew_subs 2023-12-15T17:24:05.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb71da9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb71da9640 1 -- 172.21.15.69:0/4230168939 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c13c570 con 0x7fcb6c0b2520 2023-12-15T17:24:05.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.091+0000 7fcb71da9640 1 librados: init done 2023-12-15T17:24:05.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.092+0000 7fcb71da9640 10 client.21463.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:05.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.092+0000 7fcb71da9640 15 client.21463.objecter _session_command_op_assign -1 1 2023-12-15T17:24:05.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.092+0000 7fcb71da9640 20 client.21463.objecter _get_session osd=-1 returning homeless 2023-12-15T17:24:05.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.092+0000 7fcb71da9640 10 client.21463.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:05.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.092+0000 7fcb71da9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.092+0000 7fcb71da9640 1 -- 172.21.15.69:0/4230168939 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcb6c0aad80 con 0x7fcb6c0b2520 2023-12-15T17:24:05.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.110+0000 7fcb63fff640 1 -- 172.21.15.69:0/4230168939 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fcb54003ca0 con 0x7fcb6c0b2520 2023-12-15T17:24:05.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.110+0000 7fcb63fff640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcb4c06cbd0 0x7fcb4c06f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.111+0000 7fcb6b7fe640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcb4c06cbd0 0x7fcb4c06f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.111+0000 7fcb6b7fe640 10 monclient: get_auth_request con 0x7fcb4c06cbd0 auth_method 0 2023-12-15T17:24:05.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.111+0000 7fcb63fff640 10 client.21463.objecter ms_handle_connect 0x7fcb4c06cbd0 2023-12-15T17:24:05.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.111+0000 7fcb6b7fe640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcb4c06cbd0 0x7fcb4c06f090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c035000 tx=0x7fcb5c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:05.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 1 -- 172.21.15.69:0/4230168939 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7fcb540588b0 con 0x7fcb6c0b2520 2023-12-15T17:24:05.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 10 client.21463.objecter ms_dispatch 0x7fcb6c000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:05.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 3 client.21463.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:05.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 3 client.21463.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:05.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 10 client.21463.objecter checking command 1 2023-12-15T17:24:05.113 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] conn(0x7fcb4c074460 0x7fcb4c076900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 20 client.21463.objecter _get_session s=0x7fcb4c073d90 osd=3 3 2023-12-15T17:24:05.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 20 client.21463.objecter put_session s=0x7fcb4c073d90 osd=3 3 2023-12-15T17:24:05.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 15 client.21463.objecter _session_command_op_remove -1 1 2023-12-15T17:24:05.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 20 client.21463.objecter _get_session s=0x7fcb4c073d90 osd=3 3 2023-12-15T17:24:05.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 20 client.21463.objecter get_session s=0x7fcb4c073d90 osd=3 3 2023-12-15T17:24:05.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 15 client.21463.objecter _session_command_op_assign 3 1 2023-12-15T17:24:05.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 20 client.21463.objecter put_session s=0x7fcb4c073d90 osd=3 4 2023-12-15T17:24:05.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 10 client.21463.objecter _send_command 1 2023-12-15T17:24:05.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 1 -- 172.21.15.69:0/4230168939 --> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb4c076fd0 con 0x7fcb4c074460 2023-12-15T17:24:05.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.112+0000 7fcb63fff640 20 client.21463.objecter dump_active .. 0 homeless 2023-12-15T17:24:05.114 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.113+0000 7fcb6bfff640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] conn(0x7fcb4c074460 0x7fcb4c076900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.115 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.113+0000 7fcb6bfff640 10 monclient: get_auth_request con 0x7fcb4c074460 auth_method 0 2023-12-15T17:24:05.115 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.115+0000 7fcb63fff640 1 -- 172.21.15.69:0/4230168939 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=908) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb540544f0 con 0x7fcb6c0b2520 2023-12-15T17:24:05.115 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.115+0000 7fcb63fff640 10 monclient: handle_get_version_reply finishing 1 version 908 2023-12-15T17:24:05.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.117+0000 7fcb63fff640 10 client.21463.objecter ms_handle_connect 0x7fcb4c074460 2023-12-15T17:24:05.117 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.117+0000 7fcb6bfff640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] conn(0x7fcb4c074460 0x7fcb4c076900 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:05.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.141+0000 7fcb63fff640 1 -- 172.21.15.69:0/4230168939 <== osd.3 v2:172.21.15.69:6824/882900488 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fcb4c076fd0 con 0x7fcb4c074460 2023-12-15T17:24:05.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.141+0000 7fcb63fff640 10 client.21463.objecter ms_dispatch 0x7fcb6c000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:24:05.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.141+0000 7fcb63fff640 10 client.21463.objecter _finish_command 1 = system:0 2023-12-15T17:24:05.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.141+0000 7fcb63fff640 20 client.21463.objecter put_session s=0x7fcb4c073d90 osd=3 4 2023-12-15T17:24:05.142 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.141+0000 7fcb63fff640 15 client.21463.objecter _session_command_op_remove 3 1 2023-12-15T17:24:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.167+0000 7fcb71da9640 10 client.21463.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-15T17:24:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.167+0000 7fcb71da9640 15 client.21463.objecter _session_command_op_assign -1 2 2023-12-15T17:24:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.167+0000 7fcb71da9640 20 client.21463.objecter _get_session s=0x7fcb4c073d90 osd=3 3 2023-12-15T17:24:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.167+0000 7fcb71da9640 20 client.21463.objecter put_session s=0x7fcb4c073d90 osd=3 3 2023-12-15T17:24:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.167+0000 7fcb71da9640 20 client.21463.objecter _get_session s=0x7fcb4c073d90 osd=3 3 2023-12-15T17:24:05.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.167+0000 7fcb71da9640 15 client.21463.objecter _session_command_op_remove -1 2 2023-12-15T17:24:05.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.167+0000 7fcb71da9640 20 client.21463.objecter get_session s=0x7fcb4c073d90 osd=3 3 2023-12-15T17:24:05.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.167+0000 7fcb71da9640 15 client.21463.objecter _session_command_op_assign 3 2 2023-12-15T17:24:05.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.167+0000 7fcb71da9640 20 client.21463.objecter put_session s=0x7fcb4c073d90 osd=3 4 2023-12-15T17:24:05.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.167+0000 7fcb71da9640 10 client.21463.objecter _send_command 2 2023-12-15T17:24:05.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.167+0000 7fcb71da9640 1 -- 172.21.15.69:0/4230168939 --> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcb6c0aaaa0 con 0x7fcb4c074460 2023-12-15T17:24:05.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.181+0000 7fcb63fff640 1 -- 172.21.15.69:0/4230168939 <== osd.3 v2:172.21.15.69:6824/882900488 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fcb64004040 con 0x7fcb4c074460 2023-12-15T17:24:05.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.181+0000 7fcb63fff640 10 client.21463.objecter ms_dispatch 0x7fcb6c000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:24:05.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.181+0000 7fcb63fff640 10 client.21463.objecter _finish_command 2 = system:0 2023-12-15T17:24:05.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.181+0000 7fcb63fff640 20 client.21463.objecter put_session s=0x7fcb4c073d90 osd=3 4 2023-12-15T17:24:05.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.181+0000 7fcb63fff640 15 client.21463.objecter _session_command_op_remove 3 2 2023-12-15T17:24:05.182 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-12-15T17:24:05.182 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-15T17:24:05.183 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-12-15T17:24:05.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 10 librados: watch_flush enter 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 10 librados: watch_flush exit 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 10 client.21463.objecter close_session for osd.3 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 1 -- 172.21.15.69:0/4230168939 >> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] conn(0x7fcb4c074460 msgr2=0x7fcb4c076900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.69:6824/882900488,v1:172.21.15.69:6825/882900488] conn(0x7fcb4c074460 0x7fcb4c076900 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 20 client.21463.objecter put_session s=0x7fcb4c073d90 osd=3 1 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 20 client.21463.objecter shutdown clearing up homeless session... 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 10 client.21463.objecter successfully canceled tick 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 1 -- 172.21.15.69:0/4230168939 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcb4c06cbd0 msgr2=0x7fcb4c06f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcb4c06cbd0 0x7fcb4c06f090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c035000 tx=0x7fcb5c030000 comp rx=0 tx=0).stop 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 10 monclient: shutdown 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 1 -- 172.21.15.69:0/4230168939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb6c0b2520 msgr2=0x7fcb6c1cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 1 --2- 172.21.15.69:0/4230168939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb6c0b2520 0x7fcb6c1cc6d0 secure :-1 s=READY pgs=5111 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400c410 tx=0x7fcb5400c8d0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb6bfff640 1 -- 172.21.15.69:0/4230168939 reap_dead start 2023-12-15T17:24:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 1 -- 172.21.15.69:0/4230168939 shutdown_connections 2023-12-15T17:24:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 1 -- 172.21.15.69:0/4230168939 >> 172.21.15.69:0/4230168939 conn(0x7fcb6c0b41f0 msgr2=0x7fcb6c0b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 1 -- 172.21.15.69:0/4230168939 shutdown_connections 2023-12-15T17:24:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 1 -- 172.21.15.69:0/4230168939 wait complete. 2023-12-15T17:24:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.182+0000 7fcb71da9640 1 librados: shutdown 2023-12-15T17:24:05.195 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-15T17:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.296+0000 7ffae2a21640 10 monclient: get_monmap_and_config 2023-12-15T17:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.296+0000 7ffae2a21640 10 monclient: build_initial_monmap 2023-12-15T17:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.296+0000 7ffae2a21640 10 monclient: monmap: 2023-12-15T17:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:05.297273+0000 2023-12-15T17:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:05.297273+0000 2023-12-15T17:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:05.297 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:05.298 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.297+0000 7ffae2a21640 1 Processor -- start 2023-12-15T17:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.297+0000 7ffae2a21640 1 -- start start 2023-12-15T17:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.297+0000 7ffae2a21640 10 monclient: init 2023-12-15T17:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-12-15T17:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffadc0a8910 0x7ffadc13ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 10 monclient(hunting): picked mon.noname-c con 0x7ffadc0a8910 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffadc0a9b70 addr v1:172.21.15.69:6789/0 2023-12-15T17:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc141f90 0x7ffadc1423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 10 monclient(hunting): picked mon.noname-b con 0x7ffadc141f90 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffadc0a8cf0 con 0x7ffadc0a9b70 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffadc0aabd0 con 0x7ffadc0a8910 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc0aaeb0 con 0x7ffadc141f90 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae2a21640 10 monclient(hunting): authenticate will time out at 2749.613631s 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.298+0000 7ffae0fbe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc141f90 0x7ffadc1423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffadbfff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffadc0a8910 0x7ffadc13ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffae0fbe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc141f90 0x7ffadc1423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42424/0 (socket says 172.21.15.69:42424) 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffae0fbe640 1 -- 172.21.15.69:0/3545693467 learned_addr learned my addr 172.21.15.69:0/3545693467 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffae0fbe640 10 monclient(hunting): get_auth_request con 0x7ffadc141f90 auth_method 0 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffae0fbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffae0fbe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffae0fbe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffadbfff640 10 monclient(hunting): get_auth_request con 0x7ffadc0a8910 auth_method 0 2023-12-15T17:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffadbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffadbfff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffadbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffae0fbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffae0fbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffae0fbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffadbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffadbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:05.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.299+0000 7ffadbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.301+0000 7ffadbfff640 10 monclient(hunting): handle_auth_done global_id 21466 payload 274 2023-12-15T17:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.301+0000 7ffadbfff640 1 -- 172.21.15.69:0/3545693467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc141f90 msgr2=0x7ffadc1423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.301+0000 7ffadbfff640 1 --2- 172.21.15.69:0/3545693467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc141f90 0x7ffadc1423f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.301+0000 7ffadbfff640 1 -- 172.21.15.69:0/3545693467 >> v1:172.21.15.69:6789/0 conn(0x7ffadc0a9b70 legacy=0x7ffadc13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.301+0000 7ffadbfff640 10 monclient: _finish_hunting 0 2023-12-15T17:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.301+0000 7ffadbfff640 1 monclient: found mon.noname-c 2023-12-15T17:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.301+0000 7ffadbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.301+0000 7ffadbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.301+0000 7ffadbfff640 1 -- 172.21.15.69:0/3545693467 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffadc0a9720 con 0x7ffadc0a8910 2023-12-15T17:24:05.303 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.303+0000 7ffadbfff640 1 --2- 172.21.15.69:0/3545693467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffadc0a8910 0x7ffadc13ec60 secure :-1 s=READY pgs=5112 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0b5310 tx=0x7ffacc02b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=174b536c4776fc69 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 1 -- 172.21.15.69:0/3545693467 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc031750 con 0x7ffadc0a8910 2023-12-15T17:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 10 monclient: dump: 2023-12-15T17:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.307 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 10 monclient: _finish_auth 0 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 10 monclient: _check_auth_tickets 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 1 -- 172.21.15.69:0/3545693467 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffacc0318b0 con 0x7ffadc0a8910 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffae2a21640 5 monclient: authenticate success, global_id 21466 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffae2a21640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffae2a21640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 1 -- 172.21.15.69:0/3545693467 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc031c40 con 0x7ffadc0a8910 2023-12-15T17:24:05.308 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffae2a21640 4 set_mon_vals no callback set 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.306+0000 7ffadaffd640 10 monclient: dump: 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.309 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.307+0000 7ffae2a21640 10 monclient: shutdown 2023-12-15T17:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.307+0000 7ffae2a21640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.307+0000 7ffae2a21640 1 -- 172.21.15.69:0/3545693467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffadc0a8910 msgr2=0x7ffadc13ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.307+0000 7ffae2a21640 1 --2- 172.21.15.69:0/3545693467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffadc0a8910 0x7ffadc13ec60 secure :-1 s=READY pgs=5112 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0b5310 tx=0x7ffacc02b7f0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.307+0000 7ffae2a21640 1 -- 172.21.15.69:0/3545693467 shutdown_connections 2023-12-15T17:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.307+0000 7ffae2a21640 1 --2- 172.21.15.69:0/3545693467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc141f90 0x7ffadc1423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.307+0000 7ffae2a21640 1 --2- 172.21.15.69:0/3545693467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffadc0a8910 0x7ffadc13ec60 unknown :-1 s=CLOSED pgs=5112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.307+0000 7ffae2a21640 1 -- 172.21.15.69:0/3545693467 >> 172.21.15.69:0/3545693467 conn(0x7ffadc0b7780 msgr2=0x7ffadc0b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.307+0000 7ffae2a21640 1 -- 172.21.15.69:0/3545693467 shutdown_connections 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.307+0000 7ffae2a21640 1 -- 172.21.15.69:0/3545693467 wait complete. 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.308+0000 7ffae2a21640 10 monclient: build_initial_monmap 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.308+0000 7ffae2a21640 10 monclient: monmap: 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:05.313 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.308+0000 7ffae2a21640 1 librados: starting msgr at 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.308+0000 7ffae2a21640 1 librados: starting objecter 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.308+0000 7ffae2a21640 1 Processor -- start 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.308+0000 7ffae2a21640 1 -- start start 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.308+0000 7ffae2a21640 1 librados: setting wanted keys 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.308+0000 7ffae2a21640 1 librados: calling monclient init 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.308+0000 7ffae2a21640 10 monclient: init 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffadc0a8910 0x7ffadc132490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 10 monclient(hunting): picked mon.noname-a con 0x7ffadc0a8910 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffadc141f90 0x7ffadc1329d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 10 monclient(hunting): picked mon.noname-b con 0x7ffadc141f90 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc140070 0x7ffadc140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 10 monclient(hunting): picked mon.noname-c con 0x7ffadc140070 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:05.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc13fb50 con 0x7ffadc0a8910 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc14c1f0 con 0x7ffadc141f90 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc140a60 con 0x7ffadc140070 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.309+0000 7ffae2a21640 10 monclient(hunting): authenticate will time out at 2749.624936s 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffae0fbe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc140070 0x7ffadc140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffadb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffadc141f90 0x7ffadc1329d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffae0fbe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc140070 0x7ffadc140520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42434/0 (socket says 172.21.15.69:42434) 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffae0fbe640 1 -- 172.21.15.69:0/3108856806 learned_addr learned my addr 172.21.15.69:0/3108856806 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffae0fbe640 10 monclient(hunting): get_auth_request con 0x7ffadc140070 auth_method 0 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffae0fbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffae0fbe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffae0fbe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffadb7fe640 10 monclient(hunting): get_auth_request con 0x7ffadc141f90 auth_method 0 2023-12-15T17:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffadb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffadb7fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.310+0000 7ffadb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.311+0000 7ffae0fbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.311+0000 7ffae0fbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.311+0000 7ffae0fbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.311+0000 7ffadb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.311+0000 7ffadb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.311+0000 7ffadb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.311+0000 7ffadbfff640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffadc0a8910 0x7ffadc132490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadbfff640 10 monclient(hunting): get_auth_request con 0x7ffadc0a8910 auth_method 0 2023-12-15T17:24:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadbfff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadb7fe640 10 monclient(hunting): handle_auth_done global_id 21469 payload 824 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadb7fe640 1 -- 172.21.15.69:0/3108856806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc140070 msgr2=0x7ffadc140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadb7fe640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc140070 0x7ffadc140520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadb7fe640 1 -- 172.21.15.69:0/3108856806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffadc0a8910 msgr2=0x7ffadc132490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadb7fe640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffadc0a8910 0x7ffadc132490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadb7fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadb7fe640 1 monclient: found mon.noname-b 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.313+0000 7ffadb7fe640 1 -- 172.21.15.69:0/3108856806 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffacc009400 con 0x7ffadc141f90 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.315+0000 7ffadb7fe640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffadc141f90 0x7ffadc1329d0 secure :-1 s=READY pgs=5113 cs=0 l=1 rev1=1 crypto rx=0x7ffad000c410 tx=0x7ffad000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.315+0000 7ffad8ff9640 10 client.?.objecter ms_handle_connect 0x7ffadc141f90 2023-12-15T17:24:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.315+0000 7ffad8ff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:05.319 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 1 -- 172.21.15.69:0/3108856806 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad0005230 con 0x7ffadc141f90 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 10 monclient: dump: 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 10 monclient: _finish_auth 0 2023-12-15T17:24:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 10 monclient: _check_auth_tickets 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 1 -- 172.21.15.69:0/3108856806 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad0005390 con 0x7ffadc141f90 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 1 -- 172.21.15.69:0/3108856806 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad0004820 con 0x7ffadc141f90 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffae2a21640 5 monclient: authenticate success, global_id 21469 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffae2a21640 10 monclient: _renew_subs 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffae2a21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffae2a21640 1 -- 172.21.15.69:0/3108856806 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffadc140c80 con 0x7ffadc141f90 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffad8ff9640 10 monclient: dump: 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.321 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffada7fc640 4 set_mon_vals no callback set 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffae2a21640 10 client.21469.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffae2a21640 10 monclient: _renew_subs 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffae2a21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffae2a21640 1 -- 172.21.15.69:0/3108856806 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffadc13a5b0 con 0x7ffadc141f90 2023-12-15T17:24:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.319+0000 7ffae2a21640 1 librados: init done 2023-12-15T17:24:05.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.320+0000 7ffae2a21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:05.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.320+0000 7ffae2a21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:05.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.320+0000 7ffae2a21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.323 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.320+0000 7ffae2a21640 1 -- 172.21.15.69:0/3108856806 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffadc0ab8c0 con 0x7ffadc141f90 2023-12-15T17:24:05.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.555+0000 7ffad8ff9640 1 -- 172.21.15.69:0/3108856806 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ffad0004980 con 0x7ffadc141f90 2023-12-15T17:24:05.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.555+0000 7ffad8ff9640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ffac806cb50 0x7ffac806f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.556+0000 7ffadbfff640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ffac806cb50 0x7ffac806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.556+0000 7ffadbfff640 10 monclient: get_auth_request con 0x7ffac806cb50 auth_method 0 2023-12-15T17:24:05.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.556+0000 7ffadbfff640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ffac806cb50 0x7ffac806f010 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ffacc035000 tx=0x7ffacc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:05.557 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.556+0000 7ffad8ff9640 10 client.21469.objecter ms_handle_connect 0x7ffac806cb50 2023-12-15T17:24:05.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.559+0000 7ffad8ff9640 1 -- 172.21.15.69:0/3108856806 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7ffad00578e0 con 0x7ffadc141f90 2023-12-15T17:24:05.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.559+0000 7ffad8ff9640 10 client.21469.objecter ms_dispatch 0x7ffadc000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:05.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.559+0000 7ffad8ff9640 3 client.21469.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:05.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.559+0000 7ffad8ff9640 3 client.21469.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:05.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.559+0000 7ffad8ff9640 20 client.21469.objecter dump_active .. 0 homeless 2023-12-15T17:24:05.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.568+0000 7ffad8ff9640 1 -- 172.21.15.69:0/3108856806 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ffad0053c20 con 0x7ffadc141f90 2023-12-15T17:24:05.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.568+0000 7ffad8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:05.569 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.568+0000 7ffad8ff9640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:05.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.744+0000 7ffae2a21640 1 -- 172.21.15.69:0/3108856806 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7ffadc13c860 con 0x7ffac806cb50 2023-12-15T17:24:05.745 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.745+0000 7ffad8ff9640 1 -- 172.21.15.69:0/3108856806 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ffadc13c860 con 0x7ffac806cb50 2023-12-15T17:24:05.745 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 3 to deep-scrub 2023-12-15T17:24:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 10 librados: watch_flush enter 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 10 librados: watch_flush exit 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 20 client.21469.objecter shutdown clearing up homeless session... 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 10 client.21469.objecter successfully canceled tick 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 -- 172.21.15.69:0/3108856806 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ffac806cb50 msgr2=0x7ffac806f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ffac806cb50 0x7ffac806f010 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ffacc035000 tx=0x7ffacc008030 comp rx=0 tx=0).stop 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 10 monclient: shutdown 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 -- 172.21.15.69:0/3108856806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffadc141f90 msgr2=0x7ffadc1329d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffadc141f90 0x7ffadc1329d0 secure :-1 s=READY pgs=5113 cs=0 l=1 rev1=1 crypto rx=0x7ffad000c410 tx=0x7ffad000c8d0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 -- 172.21.15.69:0/3108856806 shutdown_connections 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ffac806cb50 0x7ffac806f010 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffadc140070 0x7ffadc140520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffadc141f90 0x7ffadc1329d0 unknown :-1 s=CLOSED pgs=5113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 --2- 172.21.15.69:0/3108856806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffadc0a8910 0x7ffadc132490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 -- 172.21.15.69:0/3108856806 >> 172.21.15.69:0/3108856806 conn(0x7ffadc0b7780 msgr2=0x7ffadc0b7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:05.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 -- 172.21.15.69:0/3108856806 shutdown_connections 2023-12-15T17:24:05.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.747+0000 7ffae2a21640 1 -- 172.21.15.69:0/3108856806 wait complete. 2023-12-15T17:24:05.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.748+0000 7ffae2a21640 1 librados: shutdown 2023-12-15T17:24:05.759 INFO:tasks.ceph:Scrubbing osd.4 2023-12-15T17:24:05.759 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-15T17:24:05.873 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.873+0000 7f841e84b640 10 monclient: get_monmap_and_config 2023-12-15T17:24:05.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.873+0000 7f841e84b640 10 monclient: build_initial_monmap 2023-12-15T17:24:05.874 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.873+0000 7f841e84b640 10 monclient: monmap: 2023-12-15T17:24:05.874 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:05.874 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.874 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:05.873949+0000 2023-12-15T17:24:05.874 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:05.873949+0000 2023-12-15T17:24:05.874 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:05.874 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.874 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:05.874 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:05.874 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:05.875 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:05.875 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:05.875 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.874+0000 7f841e84b640 1 Processor -- start 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.874+0000 7f841e84b640 1 -- start start 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.874+0000 7f841e84b640 10 monclient: init 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.874+0000 7f841e84b640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.874+0000 7f841e84b640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.874+0000 7f841e84b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84180ad820 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841e84b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8418071ed0 0x7f841813e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841e84b640 10 monclient(hunting): picked mon.noname-a con 0x7f8418071ed0 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841e84b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8418070770 0x7f841813ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841e84b640 10 monclient(hunting): picked mon.noname-b con 0x7f8418070770 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841e84b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f841813f440 con 0x7f84180ad820 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841e84b640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841e84b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8418071840 con 0x7f8418071ed0 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841e84b640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841e84b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841806f750 con 0x7f8418070770 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841e84b640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841e84b640 10 monclient(hunting): authenticate will time out at 2750.190446s 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f841cde8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8418070770 0x7f841813ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f84177fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8418071ed0 0x7f841813e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f8417fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f84180ad820 0x7f84180ae050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.69:45474/0 (socket says 172.21.15.69:45474) 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.875+0000 7f8417fff640 1 -- 172.21.15.69:0/2546206678 learned_addr learned my addr 172.21.15.69:0/2546206678 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f841cde8640 10 monclient(hunting): get_auth_request con 0x7f8418070770 auth_method 0 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f841cde8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f841cde8640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f841cde8640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f84177fe640 10 monclient(hunting): get_auth_request con 0x7f8418071ed0 auth_method 0 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f84177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f84177fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f84177fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f841cde8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:05.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f841cde8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:05.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f841cde8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:05.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f84177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:05.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f84177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:05.879 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.876+0000 7f84177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.878+0000 7f8416ffd640 1 -- 172.21.15.69:0/2546206678 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466621042 0 0) 0x7f841813f440 con 0x7f84180ad820 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.878+0000 7f8416ffd640 10 monclient(hunting): _negotiate 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.878+0000 7f8416ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.878+0000 7f8416ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.878+0000 7f8416ffd640 10 monclient(hunting): my global_id is 21472 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.878+0000 7f8416ffd640 1 -- 172.21.15.69:0/2546206678 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8400003520 con 0x7f84180ad820 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.878+0000 7f841cde8640 10 monclient(hunting): handle_auth_done global_id 17822 payload 274 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.878+0000 7f841cde8640 1 -- 172.21.15.69:0/2546206678 >> v2:172.21.15.69:3300/0 conn(0x7f8418071ed0 msgr2=0x7f841813e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.879+0000 7f841cde8640 1 --2- 172.21.15.69:0/2546206678 >> v2:172.21.15.69:3300/0 conn(0x7f8418071ed0 0x7f841813e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.879+0000 7f841cde8640 1 -- 172.21.15.69:0/2546206678 >> v1:172.21.15.192:6789/0 conn(0x7f84180ad820 legacy=0x7f84180ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.879+0000 7f841cde8640 10 monclient: _finish_hunting 0 2023-12-15T17:24:05.880 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.879+0000 7f841cde8640 1 monclient: found mon.noname-b 2023-12-15T17:24:05.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.879+0000 7f841cde8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:05.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.879+0000 7f841cde8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:24:05.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.879+0000 7f841cde8640 1 -- 172.21.15.69:0/2546206678 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8418071ab0 con 0x7f8418070770 2023-12-15T17:24:05.881 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.881+0000 7f841cde8640 1 --2- 172.21.15.69:0/2546206678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8418070770 0x7f841813ef00 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7f8410009d90 tx=0x7f841002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aa64f0a4e859586 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:05.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 1 -- 172.21.15.69:0/2546206678 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841002cc50 con 0x7f8418070770 2023-12-15T17:24:05.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 10 monclient: dump: 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 10 monclient: _finish_auth 0 2023-12-15T17:24:05.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 1 -- 172.21.15.69:0/2546206678 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841002cdb0 con 0x7f8418070770 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 1 -- 172.21.15.69:0/2546206678 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410034660 con 0x7f8418070770 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f8416ffd640 10 monclient: dump: 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.888 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f841e84b640 5 monclient: authenticate success, global_id 17822 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f841e84b640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.884+0000 7f841e84b640 4 set_mon_vals no callback set 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.885+0000 7f841e84b640 10 monclient: shutdown 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.885+0000 7f841e84b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.885+0000 7f841e84b640 1 -- 172.21.15.69:0/2546206678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8418070770 msgr2=0x7f841813ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.885+0000 7f841e84b640 1 --2- 172.21.15.69:0/2546206678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8418070770 0x7f841813ef00 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7f8410009d90 tx=0x7f841002bea0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.885+0000 7f841e84b640 1 -- 172.21.15.69:0/2546206678 shutdown_connections 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.885+0000 7f841e84b640 1 --2- 172.21.15.69:0/2546206678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8418070770 0x7f841813ef00 unknown :-1 s=CLOSED pgs=3134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.885+0000 7f841e84b640 1 --2- 172.21.15.69:0/2546206678 >> v2:172.21.15.69:3300/0 conn(0x7f8418071ed0 0x7f841813e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.885+0000 7f841e84b640 1 -- 172.21.15.69:0/2546206678 >> 172.21.15.69:0/2546206678 conn(0x7f84180b3650 msgr2=0x7f84180b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.885+0000 7f841e84b640 1 -- 172.21.15.69:0/2546206678 shutdown_connections 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.885+0000 7f841e84b640 1 -- 172.21.15.69:0/2546206678 wait complete. 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.886+0000 7f841e84b640 10 monclient: build_initial_monmap 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.886+0000 7f841e84b640 10 monclient: monmap: 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.886+0000 7f841e84b640 1 librados: starting msgr at 2023-12-15T17:24:05.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.886+0000 7f841e84b640 1 librados: starting objecter 2023-12-15T17:24:05.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.886+0000 7f841e84b640 1 Processor -- start 2023-12-15T17:24:05.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.886+0000 7f841e84b640 1 -- start start 2023-12-15T17:24:05.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.886+0000 7f841e84b640 1 librados: setting wanted keys 2023-12-15T17:24:05.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.886+0000 7f841e84b640 1 librados: calling monclient init 2023-12-15T17:24:05.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.886+0000 7f841e84b640 10 monclient: init 2023-12-15T17:24:05.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:05.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-15T17:24:05.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8418070770 0x7f84180acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 10 monclient(hunting): picked mon.noname-b con 0x7f8418070770 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:05.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8418071ed0 0x7f8418138f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 10 monclient(hunting): picked mon.noname-c con 0x7f8418071ed0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:05.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84180ad820 0x7f841813b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 10 monclient(hunting): picked mon.noname-a con 0x7f84180ad820 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:05.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841813afa0 con 0x7f84180ad820 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418139450 con 0x7f8418070770 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84181422b0 con 0x7f8418071ed0 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f841e84b640 10 monclient(hunting): authenticate will time out at 2750.202857s 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.887+0000 7f84177fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8418071ed0 0x7f8418138f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f8417fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8418070770 0x7f84180acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f841cde8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84180ad820 0x7f841813b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f84177fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8418071ed0 0x7f8418138f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42452/0 (socket says 172.21.15.69:42452) 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f84177fe640 1 -- 172.21.15.69:0/3756554491 learned_addr learned my addr 172.21.15.69:0/3756554491 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f84177fe640 10 monclient(hunting): get_auth_request con 0x7f8418071ed0 auth_method 0 2023-12-15T17:24:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f84177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f84177fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f84177fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f8417fff640 10 monclient(hunting): get_auth_request con 0x7f8418070770 auth_method 0 2023-12-15T17:24:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f8417fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f8417fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f8417fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f841cde8640 10 monclient(hunting): get_auth_request con 0x7f84180ad820 auth_method 0 2023-12-15T17:24:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f841cde8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f841cde8640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f841cde8640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f8417fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f8417fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.888+0000 7f8417fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.889+0000 7f84177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.889+0000 7f84177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.889+0000 7f84177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.889+0000 7f841cde8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.889+0000 7f841cde8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.889+0000 7f841cde8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.891+0000 7f8417fff640 10 monclient(hunting): handle_auth_done global_id 21475 payload 824 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.891+0000 7f8417fff640 1 -- 172.21.15.69:0/3756554491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8418071ed0 msgr2=0x7f8418138f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.891+0000 7f8417fff640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8418071ed0 0x7f8418138f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.891+0000 7f8417fff640 1 -- 172.21.15.69:0/3756554491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84180ad820 msgr2=0x7f841813b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.891+0000 7f8417fff640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84180ad820 0x7f841813b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.891+0000 7f8417fff640 10 monclient: _finish_hunting 0 2023-12-15T17:24:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.891+0000 7f8417fff640 1 monclient: found mon.noname-b 2023-12-15T17:24:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.891+0000 7f8417fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.891+0000 7f8417fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.891+0000 7f8417fff640 1 -- 172.21.15.69:0/3756554491 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8410009a70 con 0x7f8418070770 2023-12-15T17:24:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.893+0000 7f8417fff640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8418070770 0x7f84180acb20 secure :-1 s=READY pgs=5115 cs=0 l=1 rev1=1 crypto rx=0x7f840800c390 tx=0x7f840800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.893+0000 7f8414ff9640 10 client.?.objecter ms_handle_connect 0x7f8418070770 2023-12-15T17:24:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.893+0000 7f8414ff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 1 -- 172.21.15.69:0/3756554491 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408005ce0 con 0x7f8418070770 2023-12-15T17:24:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 10 monclient: dump: 2023-12-15T17:24:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 10 monclient: _finish_auth 0 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 10 monclient: _check_auth_tickets 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f841e84b640 5 monclient: authenticate success, global_id 21475 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 1 -- 172.21.15.69:0/3756554491 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8408005e40 con 0x7f8418070770 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f841e84b640 10 monclient: _renew_subs 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f841e84b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f841e84b640 1 -- 172.21.15.69:0/3756554491 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841813b1a0 con 0x7f8418070770 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 1 -- 172.21.15.69:0/3756554491 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84080072d0 con 0x7f8418070770 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:05.902 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8414ff9640 10 monclient: dump: 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f841e84b640 10 client.21475.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f841e84b640 10 monclient: _renew_subs 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f841e84b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.903 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f841e84b640 1 -- 172.21.15.69:0/3756554491 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841813be40 con 0x7f8418070770 2023-12-15T17:24:05.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f8415ffb640 4 set_mon_vals no callback set 2023-12-15T17:24:05.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.897+0000 7f841e84b640 1 librados: init done 2023-12-15T17:24:05.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.898+0000 7f841e84b640 10 client.21475.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:05.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.898+0000 7f841e84b640 15 client.21475.objecter _session_command_op_assign -1 1 2023-12-15T17:24:05.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.898+0000 7f841e84b640 20 client.21475.objecter _get_session osd=-1 returning homeless 2023-12-15T17:24:05.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.898+0000 7f841e84b640 10 client.21475.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:05.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.898+0000 7f841e84b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:05.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.898+0000 7f841e84b640 1 -- 172.21.15.69:0/3756554491 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f84181492d0 con 0x7f8418070770 2023-12-15T17:24:05.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.916+0000 7f8414ff9640 1 -- 172.21.15.69:0/3756554491 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8408007430 con 0x7f8418070770 2023-12-15T17:24:05.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.916+0000 7f8414ff9640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f840406cb80 0x7f840406f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.917+0000 7f84177fe640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f840406cb80 0x7f840406f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.917+0000 7f84177fe640 10 monclient: get_auth_request con 0x7f840406cb80 auth_method 0 2023-12-15T17:24:05.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.917+0000 7f84177fe640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f840406cb80 0x7f840406f040 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8418139d30 tx=0x7f840c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:05.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.917+0000 7f8414ff9640 10 client.21475.objecter ms_handle_connect 0x7f840406cb80 2023-12-15T17:24:05.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 1 -- 172.21.15.69:0/3756554491 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7f84080583a0 con 0x7f8418070770 2023-12-15T17:24:05.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 10 client.21475.objecter ms_dispatch 0x7f8418000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:05.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 3 client.21475.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:05.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 3 client.21475.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:05.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 10 client.21475.objecter checking command 1 2023-12-15T17:24:05.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] conn(0x7f8404074410 0x7f84040768b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:05.920 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 20 client.21475.objecter _get_session s=0x7f8404073d40 osd=4 3 2023-12-15T17:24:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 20 client.21475.objecter put_session s=0x7f8404073d40 osd=4 3 2023-12-15T17:24:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 15 client.21475.objecter _session_command_op_remove -1 1 2023-12-15T17:24:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 20 client.21475.objecter _get_session s=0x7f8404073d40 osd=4 3 2023-12-15T17:24:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 20 client.21475.objecter get_session s=0x7f8404073d40 osd=4 3 2023-12-15T17:24:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 15 client.21475.objecter _session_command_op_assign 4 1 2023-12-15T17:24:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 20 client.21475.objecter put_session s=0x7f8404073d40 osd=4 4 2023-12-15T17:24:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 10 client.21475.objecter _send_command 1 2023-12-15T17:24:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 1 -- 172.21.15.69:0/3756554491 --> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8404076f80 con 0x7f8404074410 2023-12-15T17:24:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 20 client.21475.objecter dump_active .. 0 homeless 2023-12-15T17:24:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 1 -- 172.21.15.69:0/3756554491 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=908) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84080546e0 con 0x7f8418070770 2023-12-15T17:24:05.921 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.919+0000 7f8414ff9640 10 monclient: handle_get_version_reply finishing 1 version 908 2023-12-15T17:24:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.921+0000 7f841cde8640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] conn(0x7f8404074410 0x7f84040768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:05.922 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.921+0000 7f841cde8640 10 monclient: get_auth_request con 0x7f8404074410 auth_method 0 2023-12-15T17:24:05.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.926+0000 7f841cde8640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] conn(0x7f8404074410 0x7f84040768b0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:05.926 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.926+0000 7f8414ff9640 10 client.21475.objecter ms_handle_connect 0x7f8404074410 2023-12-15T17:24:05.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.949+0000 7f8414ff9640 1 -- 172.21.15.69:0/3756554491 <== osd.4 v2:172.21.15.192:6802/1420550374 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f8404076f80 con 0x7f8404074410 2023-12-15T17:24:05.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.949+0000 7f8414ff9640 10 client.21475.objecter ms_dispatch 0x7f8418000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:24:05.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.949+0000 7f8414ff9640 10 client.21475.objecter _finish_command 1 = system:0 2023-12-15T17:24:05.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.949+0000 7f8414ff9640 20 client.21475.objecter put_session s=0x7f8404073d40 osd=4 4 2023-12-15T17:24:05.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.949+0000 7f8414ff9640 15 client.21475.objecter _session_command_op_remove 4 1 2023-12-15T17:24:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.962+0000 7f841e84b640 10 client.21475.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-15T17:24:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.962+0000 7f841e84b640 15 client.21475.objecter _session_command_op_assign -1 2 2023-12-15T17:24:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.962+0000 7f841e84b640 20 client.21475.objecter _get_session s=0x7f8404073d40 osd=4 3 2023-12-15T17:24:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.962+0000 7f841e84b640 20 client.21475.objecter put_session s=0x7f8404073d40 osd=4 3 2023-12-15T17:24:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.962+0000 7f841e84b640 20 client.21475.objecter _get_session s=0x7f8404073d40 osd=4 3 2023-12-15T17:24:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.962+0000 7f841e84b640 15 client.21475.objecter _session_command_op_remove -1 2 2023-12-15T17:24:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.962+0000 7f841e84b640 20 client.21475.objecter get_session s=0x7f8404073d40 osd=4 3 2023-12-15T17:24:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.962+0000 7f841e84b640 15 client.21475.objecter _session_command_op_assign 4 2 2023-12-15T17:24:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.962+0000 7f841e84b640 20 client.21475.objecter put_session s=0x7f8404073d40 osd=4 4 2023-12-15T17:24:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.962+0000 7f841e84b640 10 client.21475.objecter _send_command 2 2023-12-15T17:24:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.962+0000 7f841e84b640 1 -- 172.21.15.69:0/3756554491 --> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f84180b3450 con 0x7f8404074410 2023-12-15T17:24:05.975 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.975+0000 7f8414ff9640 1 -- 172.21.15.69:0/3756554491 <== osd.4 v2:172.21.15.192:6802/1420550374 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f84180b3450 con 0x7f8404074410 2023-12-15T17:24:05.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.975+0000 7f8414ff9640 10 client.21475.objecter ms_dispatch 0x7f8418000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:24:05.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.975+0000 7f8414ff9640 10 client.21475.objecter _finish_command 2 = system:0 2023-12-15T17:24:05.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.975+0000 7f8414ff9640 20 client.21475.objecter put_session s=0x7f8404073d40 osd=4 4 2023-12-15T17:24:05.976 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.975+0000 7f8414ff9640 15 client.21475.objecter _session_command_op_remove 4 2 2023-12-15T17:24:05.976 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-12-15T17:24:05.976 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-15T17:24:05.976 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-12-15T17:24:05.977 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.975+0000 7f841e84b640 10 librados: watch_flush enter 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.975+0000 7f841e84b640 10 librados: watch_flush exit 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.975+0000 7f841e84b640 10 client.21475.objecter close_session for osd.4 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.975+0000 7f841e84b640 1 -- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] conn(0x7f8404074410 msgr2=0x7f84040768b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.975+0000 7f841e84b640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:6802/1420550374,v1:172.21.15.192:6803/1420550374] conn(0x7f8404074410 0x7f84040768b0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 20 client.21475.objecter put_session s=0x7f8404073d40 osd=4 1 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 20 client.21475.objecter shutdown clearing up homeless session... 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 10 client.21475.objecter successfully canceled tick 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 1 -- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f840406cb80 msgr2=0x7f840406f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f840406cb80 0x7f840406f040 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8418139d30 tx=0x7f840c009190 comp rx=0 tx=0).stop 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 10 monclient: shutdown 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 1 -- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8418070770 msgr2=0x7f84180acb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 1 --2- 172.21.15.69:0/3756554491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8418070770 0x7f84180acb20 secure :-1 s=READY pgs=5115 cs=0 l=1 rev1=1 crypto rx=0x7f840800c390 tx=0x7f840800c850 comp rx=0 tx=0).stop 2023-12-15T17:24:05.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841cde8640 1 -- 172.21.15.69:0/3756554491 reap_dead start 2023-12-15T17:24:05.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 1 -- 172.21.15.69:0/3756554491 shutdown_connections 2023-12-15T17:24:05.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 1 -- 172.21.15.69:0/3756554491 >> 172.21.15.69:0/3756554491 conn(0x7f84180b3650 msgr2=0x7f84180b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:05.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 1 -- 172.21.15.69:0/3756554491 shutdown_connections 2023-12-15T17:24:05.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 1 -- 172.21.15.69:0/3756554491 wait complete. 2023-12-15T17:24:05.979 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:05.976+0000 7f841e84b640 1 librados: shutdown 2023-12-15T17:24:05.986 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-15T17:24:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.104+0000 7fcb22b25640 10 monclient: get_monmap_and_config 2023-12-15T17:24:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.104+0000 7fcb22b25640 10 monclient: build_initial_monmap 2023-12-15T17:24:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.104+0000 7fcb22b25640 10 monclient: monmap: 2023-12-15T17:24:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:06.104527+0000 2023-12-15T17:24:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:06.104527+0000 2023-12-15T17:24:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.104+0000 7fcb22b25640 1 Processor -- start 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.104+0000 7fcb22b25640 1 -- start start 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.104+0000 7fcb22b25640 10 monclient: init 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-12-15T17:24:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb1c0a8a30 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcb1c142bf0 0x7fcb1c13ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 10 monclient(hunting): picked mon.noname-c con 0x7fcb1c142bf0 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcb1c0a8e10 0x7fcb1c13f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 10 monclient(hunting): picked mon.noname-a con 0x7fcb1c0a8e10 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c0aac10 con 0x7fcb1c0a8a30 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb1c0aaa10 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb1c0aaef0 con 0x7fcb1c142bf0 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.105+0000 7fcb22b25640 10 monclient(hunting): authenticate will time out at 2750.420792s 2023-12-15T17:24:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.106+0000 7fcb1bfff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcb1c142bf0 0x7fcb1c13ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.106+0000 7fcb1bfff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcb1c142bf0 0x7fcb1c13ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:44014/0 (socket says 172.21.15.69:44014) 2023-12-15T17:24:06.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.106+0000 7fcb1bfff640 1 -- 172.21.15.69:0/1376839896 learned_addr learned my addr 172.21.15.69:0/1376839896 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.106+0000 7fcb1bfff640 10 monclient(hunting): get_auth_request con 0x7fcb1c142bf0 auth_method 0 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.106+0000 7fcb1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.106+0000 7fcb1bfff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.106+0000 7fcb1bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.107+0000 7fcb1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.107+0000 7fcb1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.107+0000 7fcb1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.109+0000 7fcb1bfff640 10 monclient(hunting): handle_auth_done global_id 21478 payload 274 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.109+0000 7fcb1bfff640 1 -- 172.21.15.69:0/1376839896 >> v2:172.21.15.69:3300/0 conn(0x7fcb1c0a8e10 msgr2=0x7fcb1c13f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.109+0000 7fcb1bfff640 1 --2- 172.21.15.69:0/1376839896 >> v2:172.21.15.69:3300/0 conn(0x7fcb1c0a8e10 0x7fcb1c13f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.109+0000 7fcb1bfff640 1 -- 172.21.15.69:0/1376839896 >> v1:172.21.15.192:6789/0 conn(0x7fcb1c0a8a30 legacy=0x7fcb1c1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.109+0000 7fcb1bfff640 10 monclient: _finish_hunting 0 2023-12-15T17:24:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.109+0000 7fcb1bfff640 1 monclient: found mon.noname-c 2023-12-15T17:24:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.109+0000 7fcb1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.109+0000 7fcb1bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:24:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.109+0000 7fcb1bfff640 1 -- 172.21.15.69:0/1376839896 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1c0a97f0 con 0x7fcb1c142bf0 2023-12-15T17:24:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.111+0000 7fcb1bfff640 1 --2- 172.21.15.69:0/1376839896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb1c142bf0 0x7fcb1c13ecf0 secure :-1 s=READY pgs=5117 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c0b87d0 tx=0x7fcb0802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8b7189b96877a1c server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:06.119 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 1 -- 172.21.15.69:0/1376839896 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0802f850 con 0x7fcb1c142bf0 2023-12-15T17:24:06.119 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.119 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:06.119 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 10 monclient: dump: 2023-12-15T17:24:06.119 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 10 monclient: _finish_auth 0 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 10 monclient: _check_auth_tickets 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:06.120 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 5 monclient: authenticate success, global_id 21478 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 1 -- 172.21.15.69:0/1376839896 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0802f9b0 con 0x7fcb1c142bf0 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 4 set_mon_vals no callback set 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 1 -- 172.21.15.69:0/1376839896 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0802fd40 con 0x7fcb1c142bf0 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb1b7fe640 10 monclient: dump: 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.121 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.122 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.122 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.122 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.122 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.122 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.122 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 10 monclient: shutdown 2023-12-15T17:24:06.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:06.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 1 -- 172.21.15.69:0/1376839896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb1c142bf0 msgr2=0x7fcb1c13ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 1 --2- 172.21.15.69:0/1376839896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb1c142bf0 0x7fcb1c13ecf0 secure :-1 s=READY pgs=5117 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c0b87d0 tx=0x7fcb0802ba30 comp rx=0 tx=0).stop 2023-12-15T17:24:06.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 1 -- 172.21.15.69:0/1376839896 shutdown_connections 2023-12-15T17:24:06.125 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 1 --2- 172.21.15.69:0/1376839896 >> v2:172.21.15.69:3300/0 conn(0x7fcb1c0a8e10 0x7fcb1c13f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 1 --2- 172.21.15.69:0/1376839896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb1c142bf0 0x7fcb1c13ecf0 unknown :-1 s=CLOSED pgs=5117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 1 -- 172.21.15.69:0/1376839896 >> 172.21.15.69:0/1376839896 conn(0x7fcb1c061a30 msgr2=0x7fcb1c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 1 -- 172.21.15.69:0/1376839896 shutdown_connections 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.119+0000 7fcb22b25640 1 -- 172.21.15.69:0/1376839896 wait complete. 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.120+0000 7fcb22b25640 10 monclient: build_initial_monmap 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.120+0000 7fcb22b25640 10 monclient: monmap: 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:06.126 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.120+0000 7fcb22b25640 1 librados: starting msgr at 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.120+0000 7fcb22b25640 1 librados: starting objecter 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.120+0000 7fcb22b25640 1 Processor -- start 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.120+0000 7fcb22b25640 1 -- start start 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.120+0000 7fcb22b25640 1 librados: setting wanted keys 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.120+0000 7fcb22b25640 1 librados: calling monclient init 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.120+0000 7fcb22b25640 10 monclient: init 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb1c0a8a30 0x7fcb1c0aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 10 monclient(hunting): picked mon.noname-a con 0x7fcb1c0a8a30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb1c0a8e10 0x7fcb1c0addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 10 monclient(hunting): picked mon.noname-b con 0x7fcb1c0a8e10 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb1c142bf0 0x7fcb1c1dab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 10 monclient(hunting): picked mon.noname-c con 0x7fcb1c142bf0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c0af0d0 con 0x7fcb1c0a8a30 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c1da620 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c0ae300 con 0x7fcb1c142bf0 2023-12-15T17:24:06.128 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.121+0000 7fcb22b25640 10 monclient(hunting): authenticate will time out at 2750.437037s 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.122+0000 7fcb208c1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb1c0a8a30 0x7fcb1c0aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.122+0000 7fcb208c1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb1c0a8a30 0x7fcb1c0aeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45462/0 (socket says 172.21.15.69:45462) 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.122+0000 7fcb208c1640 1 -- 172.21.15.69:0/4163581642 learned_addr learned my addr 172.21.15.69:0/4163581642 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.122+0000 7fcb208c1640 10 monclient(hunting): get_auth_request con 0x7fcb1c0a8a30 auth_method 0 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.122+0000 7fcb208c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.122+0000 7fcb208c1640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.122+0000 7fcb208c1640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.123+0000 7fcb1bfff640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb1c0a8e10 0x7fcb1c0addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.123+0000 7fcb1bfff640 10 monclient(hunting): get_auth_request con 0x7fcb1c0a8e10 auth_method 0 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.123+0000 7fcb1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.129 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.123+0000 7fcb1bfff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.123+0000 7fcb1bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.124+0000 7fcb1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.124+0000 7fcb1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.124+0000 7fcb1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.124+0000 7fcb208c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.124+0000 7fcb208c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.130 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.124+0000 7fcb208c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.132 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.126+0000 7fcb1bfff640 10 monclient(hunting): handle_auth_done global_id 21481 payload 824 2023-12-15T17:24:06.132 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.126+0000 7fcb1bfff640 1 -- 172.21.15.69:0/4163581642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb1c142bf0 msgr2=0x7fcb1c1dab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.132 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.126+0000 7fcb1bfff640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb1c142bf0 0x7fcb1c1dab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.126+0000 7fcb1bfff640 1 -- 172.21.15.69:0/4163581642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb1c0a8a30 msgr2=0x7fcb1c0aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.126+0000 7fcb1bfff640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb1c0a8a30 0x7fcb1c0aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.126+0000 7fcb1bfff640 10 monclient: _finish_hunting 0 2023-12-15T17:24:06.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.126+0000 7fcb1bfff640 1 monclient: found mon.noname-b 2023-12-15T17:24:06.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.126+0000 7fcb1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:06.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.126+0000 7fcb1bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:06.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.126+0000 7fcb1bfff640 1 -- 172.21.15.69:0/4163581642 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb080094d0 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.129+0000 7fcb1bfff640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb1c0a8e10 0x7fcb1c0addc0 secure :-1 s=READY pgs=5118 cs=0 l=1 rev1=1 crypto rx=0x7fcb08033000 tx=0x7fcb08004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:06.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.129+0000 7fcb197fa640 10 client.?.objecter ms_handle_connect 0x7fcb1c0a8e10 2023-12-15T17:24:06.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.129+0000 7fcb197fa640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:06.133 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.130+0000 7fcb197fa640 1 -- 172.21.15.69:0/4163581642 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0803c040 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.130+0000 7fcb197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.130+0000 7fcb197fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.130+0000 7fcb197fa640 10 monclient: dump: 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.134 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.130+0000 7fcb197fa640 10 monclient: _finish_auth 0 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.130+0000 7fcb197fa640 10 monclient: _check_auth_tickets 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.130+0000 7fcb197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.130+0000 7fcb22b25640 5 monclient: authenticate success, global_id 21481 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.130+0000 7fcb22b25640 10 monclient: _renew_subs 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.130+0000 7fcb22b25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.130+0000 7fcb22b25640 1 -- 172.21.15.69:0/4163581642 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1c1db080 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.131+0000 7fcb22b25640 10 client.21481.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.131+0000 7fcb22b25640 10 monclient: _renew_subs 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.131+0000 7fcb22b25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.131+0000 7fcb22b25640 1 -- 172.21.15.69:0/4163581642 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1c0b0640 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.131+0000 7fcb22b25640 1 librados: init done 2023-12-15T17:24:06.135 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.131+0000 7fcb22b25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.131+0000 7fcb22b25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.131+0000 7fcb22b25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.131+0000 7fcb22b25640 1 -- 172.21.15.69:0/4163581642 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1c0af750 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.132+0000 7fcb197fa640 1 -- 172.21.15.69:0/4163581642 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0804c440 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.132+0000 7fcb197fa640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.132+0000 7fcb197fa640 1 -- 172.21.15.69:0/4163581642 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0804c7b0 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.132+0000 7fcb197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.132+0000 7fcb197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.132+0000 7fcb197fa640 10 monclient: dump: 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.136 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.137 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.137 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.137 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.137 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.137 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.137 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.137 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.132+0000 7fcb1a7fc640 4 set_mon_vals no callback set 2023-12-15T17:24:06.376 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.376+0000 7fcb197fa640 1 -- 172.21.15.69:0/4163581642 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fcb080423f0 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.376+0000 7fcb197fa640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcb0006cb50 0x7fcb0006f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.376+0000 7fcb208c1640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcb0006cb50 0x7fcb0006f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.376+0000 7fcb208c1640 10 monclient: get_auth_request con 0x7fcb0006cb50 auth_method 0 2023-12-15T17:24:06.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.377+0000 7fcb208c1640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcb0006cb50 0x7fcb0006f010 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcb10007fd0 tx=0x7fcb10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:06.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.377+0000 7fcb197fa640 10 client.21481.objecter ms_handle_connect 0x7fcb0006cb50 2023-12-15T17:24:06.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.380+0000 7fcb197fa640 1 -- 172.21.15.69:0/4163581642 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7fcb0807ace0 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.380+0000 7fcb197fa640 10 client.21481.objecter ms_dispatch 0x7fcb1c000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:06.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.380+0000 7fcb197fa640 3 client.21481.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:06.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.380+0000 7fcb197fa640 3 client.21481.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:06.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.380+0000 7fcb197fa640 20 client.21481.objecter dump_active .. 0 homeless 2023-12-15T17:24:06.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.390+0000 7fcb197fa640 1 -- 172.21.15.69:0/4163581642 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fcb08077020 con 0x7fcb1c0a8e10 2023-12-15T17:24:06.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.390+0000 7fcb197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:06.390 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.390+0000 7fcb197fa640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:06.558 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.557+0000 7fcb22b25640 1 -- 172.21.15.69:0/4163581642 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb1c13f770 con 0x7fcb0006cb50 2023-12-15T17:24:06.558 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.558+0000 7fcb197fa640 1 -- 172.21.15.69:0/4163581642 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcb1000f850 con 0x7fcb0006cb50 2023-12-15T17:24:06.558 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 4 to deep-scrub 2023-12-15T17:24:06.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 10 librados: watch_flush enter 2023-12-15T17:24:06.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 10 librados: watch_flush exit 2023-12-15T17:24:06.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 20 client.21481.objecter shutdown clearing up homeless session... 2023-12-15T17:24:06.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 10 client.21481.objecter successfully canceled tick 2023-12-15T17:24:06.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 -- 172.21.15.69:0/4163581642 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcb0006cb50 msgr2=0x7fcb0006f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcb0006cb50 0x7fcb0006f010 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcb10007fd0 tx=0x7fcb10009040 comp rx=0 tx=0).stop 2023-12-15T17:24:06.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 10 monclient: shutdown 2023-12-15T17:24:06.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:06.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 -- 172.21.15.69:0/4163581642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb1c0a8e10 msgr2=0x7fcb1c0addc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb1c0a8e10 0x7fcb1c0addc0 secure :-1 s=READY pgs=5118 cs=0 l=1 rev1=1 crypto rx=0x7fcb08033000 tx=0x7fcb08004730 comp rx=0 tx=0).stop 2023-12-15T17:24:06.561 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 -- 172.21.15.69:0/4163581642 shutdown_connections 2023-12-15T17:24:06.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fcb0006cb50 0x7fcb0006f010 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb1c142bf0 0x7fcb1c1dab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb1c0a8e10 0x7fcb1c0addc0 unknown :-1 s=CLOSED pgs=5118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 --2- 172.21.15.69:0/4163581642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb1c0a8a30 0x7fcb1c0aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 -- 172.21.15.69:0/4163581642 >> 172.21.15.69:0/4163581642 conn(0x7fcb1c061a30 msgr2=0x7fcb1c13cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:06.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 -- 172.21.15.69:0/4163581642 shutdown_connections 2023-12-15T17:24:06.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 -- 172.21.15.69:0/4163581642 wait complete. 2023-12-15T17:24:06.562 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.560+0000 7fcb22b25640 1 librados: shutdown 2023-12-15T17:24:06.571 INFO:tasks.ceph:Scrubbing osd.5 2023-12-15T17:24:06.572 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-15T17:24:06.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.674+0000 7fc810290640 10 monclient: get_monmap_and_config 2023-12-15T17:24:06.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.674+0000 7fc810290640 10 monclient: build_initial_monmap 2023-12-15T17:24:06.675 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.674+0000 7fc810290640 10 monclient: monmap: 2023-12-15T17:24:06.675 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:06.675 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.675 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:06.674896+0000 2023-12-15T17:24:06.675 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:06.674896+0000 2023-12-15T17:24:06.675 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:06.675 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.676 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:06.676 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:06.676 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:06.676 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:06.676 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:06.676 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.674+0000 7fc810290640 1 Processor -- start 2023-12-15T17:24:06.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 1 -- start start 2023-12-15T17:24:06.676 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 10 monclient: init 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc8080ad820 0x7fc8080ae050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 10 monclient(hunting): picked mon.noname-a con 0x7fc8080ad820 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc808071ed0 addr v1:172.21.15.69:6789/0 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 0x7fc8081492d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 10 monclient(hunting): picked mon.noname-b con 0x7fc808070770 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc808071840 con 0x7fc808071ed0 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc80806f750 con 0x7fc8080ad820 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc808071ab0 con 0x7fc808070770 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:06.678 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.675+0000 7fc810290640 10 monclient(hunting): authenticate will time out at 2750.991153s 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e02c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc8080ad820 0x7fc8080ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e82d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 0x7fc8081492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e82d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 0x7fc8081492d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42470/0 (socket says 172.21.15.69:42470) 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e82d640 1 -- 172.21.15.69:0/1451927613 learned_addr learned my addr 172.21.15.69:0/1451927613 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e02c640 10 monclient(hunting): get_auth_request con 0x7fc8080ad820 auth_method 0 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e02c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e02c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e02c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e82d640 10 monclient(hunting): get_auth_request con 0x7fc808070770 auth_method 0 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e82d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e82d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.676+0000 7fc80e82d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.677+0000 7fc80e82d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.677+0000 7fc80e82d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.679 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.677+0000 7fc80e82d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.677+0000 7fc80e02c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.677+0000 7fc80e02c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.680 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.677+0000 7fc80e02c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.679+0000 7fc80e82d640 10 monclient(hunting): handle_auth_done global_id 17828 payload 274 2023-12-15T17:24:06.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.679+0000 7fc80e82d640 1 -- 172.21.15.69:0/1451927613 >> v2:172.21.15.69:3300/0 conn(0x7fc8080ad820 msgr2=0x7fc8080ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:06.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.679+0000 7fc80e82d640 1 --2- 172.21.15.69:0/1451927613 >> v2:172.21.15.69:3300/0 conn(0x7fc8080ad820 0x7fc8080ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.679+0000 7fc80e82d640 1 -- 172.21.15.69:0/1451927613 >> v1:172.21.15.69:6789/0 conn(0x7fc808071ed0 legacy=0x7fc80813e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:06.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.679+0000 7fc80e82d640 10 monclient: _finish_hunting 0 2023-12-15T17:24:06.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.679+0000 7fc80e82d640 1 monclient: found mon.noname-b 2023-12-15T17:24:06.681 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.679+0000 7fc80e82d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:06.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.679+0000 7fc80e82d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:24:06.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.679+0000 7fc80e82d640 1 -- 172.21.15.69:0/1451927613 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8080ae590 con 0x7fc808070770 2023-12-15T17:24:06.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.679+0000 7fc80e02c640 1 --2- 172.21.15.69:0/1451927613 >> v2:172.21.15.69:3300/0 conn(0x7fc8080ad820 0x7fc8080ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-15T17:24:06.682 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.681+0000 7fc80e82d640 1 --2- 172.21.15.69:0/1451927613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 0x7fc8081492d0 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7fc8080b44a0 tx=0x7fc80002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d620c8e9ac4d3dc6 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 1 -- 172.21.15.69:0/1451927613 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80002ca50 con 0x7fc808070770 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 10 monclient: dump: 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.685 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 10 monclient: _finish_auth 0 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 10 monclient: _check_auth_tickets 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 1 -- 172.21.15.69:0/1451927613 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80002cbb0 con 0x7fc808070770 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc810290640 5 monclient: authenticate success, global_id 17828 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc810290640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc810290640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 1 -- 172.21.15.69:0/1451927613 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8000343f0 con 0x7fc808070770 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc810290640 4 set_mon_vals no callback set 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.684+0000 7fc80d02a640 10 monclient: dump: 2023-12-15T17:24:06.686 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.687 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.687 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.687 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.687 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.687 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.687 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.687 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.687 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.687 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.685+0000 7fc810290640 10 monclient: shutdown 2023-12-15T17:24:06.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.685+0000 7fc810290640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:06.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.685+0000 7fc810290640 1 -- 172.21.15.69:0/1451927613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 msgr2=0x7fc8081492d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.685+0000 7fc810290640 1 --2- 172.21.15.69:0/1451927613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 0x7fc8081492d0 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7fc8080b44a0 tx=0x7fc80002beb0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.685+0000 7fc810290640 1 -- 172.21.15.69:0/1451927613 shutdown_connections 2023-12-15T17:24:06.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.685+0000 7fc810290640 1 --2- 172.21.15.69:0/1451927613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 0x7fc8081492d0 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.685+0000 7fc810290640 1 --2- 172.21.15.69:0/1451927613 >> v2:172.21.15.69:3300/0 conn(0x7fc8080ad820 0x7fc8080ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.685+0000 7fc810290640 1 -- 172.21.15.69:0/1451927613 >> 172.21.15.69:0/1451927613 conn(0x7fc8080b3650 msgr2=0x7fc8080b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.685+0000 7fc810290640 1 -- 172.21.15.69:0/1451927613 shutdown_connections 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.685+0000 7fc810290640 1 -- 172.21.15.69:0/1451927613 wait complete. 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.686+0000 7fc810290640 10 monclient: build_initial_monmap 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.686+0000 7fc810290640 10 monclient: monmap: 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.686+0000 7fc810290640 1 librados: starting msgr at 2023-12-15T17:24:06.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.686+0000 7fc810290640 1 librados: starting objecter 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.686+0000 7fc810290640 1 Processor -- start 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.686+0000 7fc810290640 1 -- start start 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.686+0000 7fc810290640 1 librados: setting wanted keys 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.686+0000 7fc810290640 1 librados: calling monclient init 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.686+0000 7fc810290640 10 monclient: init 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 0x7fc8081364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 10 monclient(hunting): picked mon.noname-c con 0x7fc808070770 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc808071ed0 0x7fc80813a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 10 monclient(hunting): picked mon.noname-a con 0x7fc808071ed0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8080ad820 0x7fc80813a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 10 monclient(hunting): picked mon.noname-b con 0x7fc8080ad820 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8081371d0 con 0x7fc808071ed0 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc808138900 con 0x7fc8080ad820 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc80813bda0 con 0x7fc808070770 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc810290640 10 monclient(hunting): authenticate will time out at 2751.002708s 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc80e02c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 0x7fc8081364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.687+0000 7fc80d82b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc808071ed0 0x7fc80813a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e02c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 0x7fc8081364a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42478/0 (socket says 172.21.15.69:42478) 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e02c640 1 -- 172.21.15.69:0/2820762446 learned_addr learned my addr 172.21.15.69:0/2820762446 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e02c640 10 monclient(hunting): get_auth_request con 0x7fc808070770 auth_method 0 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e02c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e02c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e02c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e82d640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8080ad820 0x7fc80813a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80d82b640 10 monclient(hunting): get_auth_request con 0x7fc808071ed0 auth_method 0 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80d82b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80d82b640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80d82b640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e82d640 10 monclient(hunting): get_auth_request con 0x7fc8080ad820 auth_method 0 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e82d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e82d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e82d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e02c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e02c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e02c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80d82b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80d82b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80d82b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e82d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e82d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.688+0000 7fc80e82d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.695 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.691+0000 7fc80d82b640 10 monclient(hunting): handle_auth_done global_id 16116 payload 824 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.691+0000 7fc80d82b640 1 -- 172.21.15.69:0/2820762446 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 msgr2=0x7fc8081364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.691+0000 7fc80d82b640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc808070770 0x7fc8081364a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.691+0000 7fc80d82b640 1 -- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8080ad820 msgr2=0x7fc80813a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.691+0000 7fc80d82b640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8080ad820 0x7fc80813a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.691+0000 7fc80d82b640 10 monclient: _finish_hunting 0 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.691+0000 7fc80d82b640 1 monclient: found mon.noname-a 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.691+0000 7fc80d82b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.691+0000 7fc80d82b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.691+0000 7fc80d82b640 1 -- 172.21.15.69:0/2820762446 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc804009400 con 0x7fc808071ed0 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.694+0000 7fc80d82b640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc808071ed0 0x7fc80813a1c0 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7fc8080722e0 tx=0x7fc7f800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.694+0000 7fc7feffd640 10 client.?.objecter ms_handle_connect 0x7fc808071ed0 2023-12-15T17:24:06.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.694+0000 7fc7feffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:06.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 1 -- 172.21.15.69:0/2820762446 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80035f0 con 0x7fc808071ed0 2023-12-15T17:24:06.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:24:06.700 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 10 monclient: dump: 2023-12-15T17:24:06.700 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.700 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 10 monclient: _finish_auth 0 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 10 monclient: _check_auth_tickets 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc810290640 5 monclient: authenticate success, global_id 16116 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 1 -- 172.21.15.69:0/2820762446 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f8003750 con 0x7fc808071ed0 2023-12-15T17:24:06.701 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 1 -- 172.21.15.69:0/2820762446 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8003b40 con 0x7fc808071ed0 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc810290640 10 monclient: _renew_subs 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc810290640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc810290640 1 -- 172.21.15.69:0/2820762446 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc800009b30 con 0x7fc808071ed0 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc7feffd640 10 monclient: dump: 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.702 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc80c829640 4 set_mon_vals no callback set 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc810290640 10 client.16116.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc810290640 10 monclient: _renew_subs 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc810290640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc810290640 1 -- 172.21.15.69:0/2820762446 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8081369e0 con 0x7fc808071ed0 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.699+0000 7fc810290640 1 librados: init done 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.700+0000 7fc810290640 10 client.16116.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.700+0000 7fc810290640 15 client.16116.objecter _session_command_op_assign -1 1 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.700+0000 7fc810290640 20 client.16116.objecter _get_session osd=-1 returning homeless 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.700+0000 7fc810290640 10 client.16116.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.700+0000 7fc810290640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:24:06.703 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.700+0000 7fc810290640 1 -- 172.21.15.69:0/2820762446 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc80813f0d0 con 0x7fc808071ed0 2023-12-15T17:24:06.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.719+0000 7fc7feffd640 1 -- 172.21.15.69:0/2820762446 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc7f8003ca0 con 0x7fc808071ed0 2023-12-15T17:24:06.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.720+0000 7fc7feffd640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc7ec06cb50 0x7fc7ec06f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.720+0000 7fc80e02c640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc7ec06cb50 0x7fc7ec06f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.720+0000 7fc80e02c640 10 monclient: get_auth_request con 0x7fc7ec06cb50 auth_method 0 2023-12-15T17:24:06.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.721+0000 7fc7feffd640 10 client.16116.objecter ms_handle_connect 0x7fc7ec06cb50 2023-12-15T17:24:06.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.721+0000 7fc80e02c640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc7ec06cb50 0x7fc7ec06f010 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc804009d30 tx=0x7fc804009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:06.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 1 -- 172.21.15.69:0/2820762446 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7fc7f8057e50 con 0x7fc808071ed0 2023-12-15T17:24:06.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 10 client.16116.objecter ms_dispatch 0x7fc808000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:06.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 3 client.16116.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:06.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 3 client.16116.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:06.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 10 client.16116.objecter checking command 1 2023-12-15T17:24:06.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] conn(0x7fc7ec0743e0 0x7fc7ec076880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 20 client.16116.objecter _get_session s=0x7fc7ec073d10 osd=5 3 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 20 client.16116.objecter put_session s=0x7fc7ec073d10 osd=5 3 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 15 client.16116.objecter _session_command_op_remove -1 1 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 20 client.16116.objecter _get_session s=0x7fc7ec073d10 osd=5 3 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 20 client.16116.objecter get_session s=0x7fc7ec073d10 osd=5 3 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 15 client.16116.objecter _session_command_op_assign 5 1 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 20 client.16116.objecter put_session s=0x7fc7ec073d10 osd=5 4 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 10 client.16116.objecter _send_command 1 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 1 -- 172.21.15.69:0/2820762446 --> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7ec076f50 con 0x7fc7ec0743e0 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 20 client.16116.objecter dump_active .. 0 homeless 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 1 -- 172.21.15.69:0/2820762446 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=908) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7f800cea0 con 0x7fc808071ed0 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.722+0000 7fc7feffd640 10 monclient: handle_get_version_reply finishing 1 version 908 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.723+0000 7fc80e82d640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] conn(0x7fc7ec0743e0 0x7fc7ec076880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.723+0000 7fc80e82d640 10 monclient: get_auth_request con 0x7fc7ec0743e0 auth_method 0 2023-12-15T17:24:06.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.726+0000 7fc7feffd640 10 client.16116.objecter ms_handle_connect 0x7fc7ec0743e0 2023-12-15T17:24:06.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.726+0000 7fc80e82d640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] conn(0x7fc7ec0743e0 0x7fc7ec076880 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:06.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.750+0000 7fc7feffd640 1 -- 172.21.15.69:0/2820762446 <== osd.5 v2:172.21.15.192:6810/2003653981 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fc7ec076f50 con 0x7fc7ec0743e0 2023-12-15T17:24:06.750 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.750+0000 7fc7feffd640 10 client.16116.objecter ms_dispatch 0x7fc808000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:24:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.750+0000 7fc7feffd640 10 client.16116.objecter _finish_command 1 = system:0 2023-12-15T17:24:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.750+0000 7fc7feffd640 20 client.16116.objecter put_session s=0x7fc7ec073d10 osd=5 4 2023-12-15T17:24:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.750+0000 7fc7feffd640 15 client.16116.objecter _session_command_op_remove 5 1 2023-12-15T17:24:06.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.765+0000 7fc810290640 10 client.16116.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-15T17:24:06.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.765+0000 7fc810290640 15 client.16116.objecter _session_command_op_assign -1 2 2023-12-15T17:24:06.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.765+0000 7fc810290640 20 client.16116.objecter _get_session s=0x7fc7ec073d10 osd=5 3 2023-12-15T17:24:06.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.765+0000 7fc810290640 20 client.16116.objecter put_session s=0x7fc7ec073d10 osd=5 3 2023-12-15T17:24:06.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.765+0000 7fc810290640 20 client.16116.objecter _get_session s=0x7fc7ec073d10 osd=5 3 2023-12-15T17:24:06.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.765+0000 7fc810290640 15 client.16116.objecter _session_command_op_remove -1 2 2023-12-15T17:24:06.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.765+0000 7fc810290640 20 client.16116.objecter get_session s=0x7fc7ec073d10 osd=5 3 2023-12-15T17:24:06.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.765+0000 7fc810290640 15 client.16116.objecter _session_command_op_assign 5 2 2023-12-15T17:24:06.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.765+0000 7fc810290640 20 client.16116.objecter put_session s=0x7fc7ec073d10 osd=5 4 2023-12-15T17:24:06.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.765+0000 7fc810290640 10 client.16116.objecter _send_command 2 2023-12-15T17:24:06.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.765+0000 7fc810290640 1 -- 172.21.15.69:0/2820762446 --> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc808075af0 con 0x7fc7ec0743e0 2023-12-15T17:24:06.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.779+0000 7fc7feffd640 1 -- 172.21.15.69:0/2820762446 <== osd.5 v2:172.21.15.192:6810/2003653981 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fc808075af0 con 0x7fc7ec0743e0 2023-12-15T17:24:06.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.779+0000 7fc7feffd640 10 client.16116.objecter ms_dispatch 0x7fc808000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:24:06.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.779+0000 7fc7feffd640 10 client.16116.objecter _finish_command 2 = system:0 2023-12-15T17:24:06.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.779+0000 7fc7feffd640 20 client.16116.objecter put_session s=0x7fc7ec073d10 osd=5 4 2023-12-15T17:24:06.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.779+0000 7fc7feffd640 15 client.16116.objecter _session_command_op_remove 5 2 2023-12-15T17:24:06.780 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-12-15T17:24:06.780 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-15T17:24:06.780 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-12-15T17:24:06.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.779+0000 7fc810290640 10 librados: watch_flush enter 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.779+0000 7fc810290640 10 librados: watch_flush exit 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.779+0000 7fc810290640 10 client.16116.objecter close_session for osd.5 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.779+0000 7fc810290640 1 -- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] conn(0x7fc7ec0743e0 msgr2=0x7fc7ec076880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.779+0000 7fc810290640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:6810/2003653981,v1:172.21.15.192:6811/2003653981] conn(0x7fc7ec0743e0 0x7fc7ec076880 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.779+0000 7fc810290640 20 client.16116.objecter put_session s=0x7fc7ec073d10 osd=5 1 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 20 client.16116.objecter shutdown clearing up homeless session... 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 10 client.16116.objecter successfully canceled tick 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 1 -- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc7ec06cb50 msgr2=0x7fc7ec06f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fc7ec06cb50 0x7fc7ec06f010 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc804009d30 tx=0x7fc804009190 comp rx=0 tx=0).stop 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 10 monclient: shutdown 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 1 -- 172.21.15.69:0/2820762446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc808071ed0 msgr2=0x7fc80813a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 1 --2- 172.21.15.69:0/2820762446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc808071ed0 0x7fc80813a1c0 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7fc8080722e0 tx=0x7fc7f800c7f0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc80e82d640 1 -- 172.21.15.69:0/2820762446 reap_dead start 2023-12-15T17:24:06.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 1 -- 172.21.15.69:0/2820762446 shutdown_connections 2023-12-15T17:24:06.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 1 -- 172.21.15.69:0/2820762446 >> 172.21.15.69:0/2820762446 conn(0x7fc8080b3650 msgr2=0x7fc8080b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:06.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 1 -- 172.21.15.69:0/2820762446 shutdown_connections 2023-12-15T17:24:06.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 1 -- 172.21.15.69:0/2820762446 wait complete. 2023-12-15T17:24:06.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.780+0000 7fc810290640 1 librados: shutdown 2023-12-15T17:24:06.791 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.891+0000 7fac19a32640 10 monclient: get_monmap_and_config 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.891+0000 7fac19a32640 10 monclient: build_initial_monmap 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.891+0000 7fac19a32640 10 monclient: monmap: 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:06.892079+0000 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:06.892079+0000 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:06.892 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:06.893 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:06.893 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.892+0000 7fac19a32640 1 Processor -- start 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.892+0000 7fac19a32640 1 -- start start 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.892+0000 7fac19a32640 10 monclient: init 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.892+0000 7fac19a32640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.892+0000 7fac19a32640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac19a32640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac140a8910 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac19a32640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac140a9b70 addr v1:172.21.15.69:6789/0 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac19a32640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fac1413f370 0x7fac141426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac19a32640 10 monclient(hunting): picked mon.noname-c con 0x7fac1413f370 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac19a32640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac140a8cf0 con 0x7fac140a9b70 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac19a32640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac140aabd0 con 0x7fac140a8910 2023-12-15T17:24:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac19a32640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac19a32640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac140aaeb0 con 0x7fac1413f370 2023-12-15T17:24:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac19a32640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac19a32640 10 monclient(hunting): authenticate will time out at 2751.208445s 2023-12-15T17:24:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac0bfff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fac140a9b70 0x7fac14141f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42730/0 (socket says 172.21.15.69:42730) 2023-12-15T17:24:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac0bfff640 1 -- 172.21.15.69:0/178798502 learned_addr learned my addr 172.21.15.69:0/178798502 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:24:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.893+0000 7fac137fe640 1 --2- 172.21.15.69:0/178798502 >> v2:172.21.15.192:3300/0 conn(0x7fac1413f370 0x7fac141426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.894+0000 7fac137fe640 10 monclient(hunting): get_auth_request con 0x7fac1413f370 auth_method 0 2023-12-15T17:24:06.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.894+0000 7fac137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.894+0000 7fac137fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.894+0000 7fac137fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.895+0000 7fac137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.895+0000 7fac137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.895+0000 7fac137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac127fc640 1 -- 172.21.15.69:0/178798502 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582298295 0 0) 0x7fac140a8cf0 con 0x7fac140a9b70 2023-12-15T17:24:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac127fc640 10 monclient(hunting): _negotiate 2023-12-15T17:24:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac127fc640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac127fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac127fc640 10 monclient(hunting): my global_id is 16119 2023-12-15T17:24:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac127fc640 1 -- 172.21.15.69:0/178798502 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac00003520 con 0x7fac140a9b70 2023-12-15T17:24:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac137fe640 10 monclient(hunting): handle_auth_done global_id 21487 payload 274 2023-12-15T17:24:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac137fe640 1 -- 172.21.15.69:0/178798502 >> v1:172.21.15.192:6789/0 conn(0x7fac140a8910 legacy=0x7fac1413ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac137fe640 1 -- 172.21.15.69:0/178798502 >> v1:172.21.15.69:6789/0 conn(0x7fac140a9b70 legacy=0x7fac14141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac137fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac137fe640 1 monclient: found mon.noname-c 2023-12-15T17:24:06.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:06.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac137fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:24:06.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.896+0000 7fac137fe640 1 -- 172.21.15.69:0/178798502 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac140a9720 con 0x7fac1413f370 2023-12-15T17:24:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.899+0000 7fac137fe640 1 --2- 172.21.15.69:0/178798502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac1413f370 0x7fac141426a0 secure :-1 s=READY pgs=5120 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009d90 tx=0x7fac0c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73c2a97e9302790d server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 1 -- 172.21.15.69:0/178798502 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c02cc60 con 0x7fac1413f370 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 10 monclient: dump: 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.904 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 10 monclient: _finish_auth 0 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 10 monclient: _check_auth_tickets 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 1 -- 172.21.15.69:0/178798502 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c02cdc0 con 0x7fac1413f370 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 1 -- 172.21.15.69:0/178798502 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c034660 con 0x7fac1413f370 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac127fc640 10 monclient: dump: 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.906 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.906 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.906 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.906 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac19a32640 5 monclient: authenticate success, global_id 21487 2023-12-15T17:24:06.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac19a32640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:06.906 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.903+0000 7fac19a32640 4 set_mon_vals no callback set 2023-12-15T17:24:06.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.904+0000 7fac19a32640 10 monclient: shutdown 2023-12-15T17:24:06.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.904+0000 7fac19a32640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:06.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.904+0000 7fac19a32640 1 -- 172.21.15.69:0/178798502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac1413f370 msgr2=0x7fac141426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.904+0000 7fac19a32640 1 --2- 172.21.15.69:0/178798502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac1413f370 0x7fac141426a0 secure :-1 s=READY pgs=5120 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009d90 tx=0x7fac0c02beb0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.904+0000 7fac19a32640 1 -- 172.21.15.69:0/178798502 shutdown_connections 2023-12-15T17:24:06.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.904+0000 7fac19a32640 1 --2- 172.21.15.69:0/178798502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac1413f370 0x7fac141426a0 unknown :-1 s=CLOSED pgs=5120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.904+0000 7fac19a32640 1 -- 172.21.15.69:0/178798502 >> 172.21.15.69:0/178798502 conn(0x7fac140b7780 msgr2=0x7fac140b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:06.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.904+0000 7fac19a32640 1 -- 172.21.15.69:0/178798502 shutdown_connections 2023-12-15T17:24:06.909 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.904+0000 7fac19a32640 1 -- 172.21.15.69:0/178798502 wait complete. 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 10 monclient: build_initial_monmap 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 10 monclient: monmap: 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 1 librados: starting msgr at 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 1 librados: starting objecter 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 1 Processor -- start 2023-12-15T17:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 1 -- start start 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 1 librados: setting wanted keys 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 1 librados: calling monclient init 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 10 monclient: init 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.905+0000 7fac19a32640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac140a8910 0x7fac1413a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 10 monclient(hunting): picked mon.noname-c con 0x7fac140a8910 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac1413f370 0x7fac1413aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 10 monclient(hunting): picked mon.noname-a con 0x7fac1413f370 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac140ac700 0x7fac1413b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 10 monclient(hunting): picked mon.noname-b con 0x7fac140ac700 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1413aa10 con 0x7fac1413f370 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac141393e0 con 0x7fac140ac700 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac140ac000 con 0x7fac140a8910 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac19a32640 10 monclient(hunting): authenticate will time out at 2751.221436s 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac1413f370 0x7fac1413aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac12ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac140a8910 0x7fac1413a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.906+0000 7fac137fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac140ac700 0x7fac1413b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac1413f370 0x7fac1413aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45498/0 (socket says 172.21.15.69:45498) 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac0bfff640 1 -- 172.21.15.69:0/286098054 learned_addr learned my addr 172.21.15.69:0/286098054 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac0bfff640 10 monclient(hunting): get_auth_request con 0x7fac1413f370 auth_method 0 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac0bfff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac0bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac137fe640 10 monclient(hunting): get_auth_request con 0x7fac140ac700 auth_method 0 2023-12-15T17:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac137fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac137fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac12ffd640 10 monclient(hunting): get_auth_request con 0x7fac140a8910 auth_method 0 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac12ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac12ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:06.913 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.907+0000 7fac12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:06.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.910+0000 7fac137fe640 10 monclient(hunting): handle_auth_done global_id 21490 payload 824 2023-12-15T17:24:06.914 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.910+0000 7fac137fe640 1 -- 172.21.15.69:0/286098054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac140a8910 msgr2=0x7fac1413a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.910+0000 7fac137fe640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac140a8910 0x7fac1413a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.910+0000 7fac137fe640 1 -- 172.21.15.69:0/286098054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac1413f370 msgr2=0x7fac1413aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.910+0000 7fac137fe640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac1413f370 0x7fac1413aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.910+0000 7fac137fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.910+0000 7fac137fe640 1 monclient: found mon.noname-b 2023-12-15T17:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.910+0000 7fac137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.910+0000 7fac137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.910+0000 7fac137fe640 1 -- 172.21.15.69:0/286098054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c009a70 con 0x7fac140ac700 2023-12-15T17:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.912+0000 7fac137fe640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac140ac700 0x7fac1413b3f0 secure :-1 s=READY pgs=5121 cs=0 l=1 rev1=1 crypto rx=0x7fac0c033000 tx=0x7fac0c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.912+0000 7fac10ff9640 10 client.?.objecter ms_handle_connect 0x7fac140ac700 2023-12-15T17:24:06.915 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.912+0000 7fac10ff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:06.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 1 -- 172.21.15.69:0/286098054 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c009450 con 0x7fac140ac700 2023-12-15T17:24:06.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.916 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 10 monclient: dump: 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 10 monclient: _finish_auth 0 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 10 monclient: _check_auth_tickets 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:06.917 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 1 -- 172.21.15.69:0/286098054 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c040450 con 0x7fac140ac700 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac19a32640 5 monclient: authenticate success, global_id 21490 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac19a32640 10 monclient: _renew_subs 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac19a32640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac19a32640 1 -- 172.21.15.69:0/286098054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac14139900 con 0x7fac140ac700 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 1 -- 172.21.15.69:0/286098054 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c0407c0 con 0x7fac140ac700 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac10ff9640 10 monclient: dump: 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:06.918 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac11ffb640 4 set_mon_vals no callback set 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac19a32640 10 client.21490.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac19a32640 10 monclient: _renew_subs 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac19a32640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac19a32640 1 -- 172.21.15.69:0/286098054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1413b930 con 0x7fac140ac700 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.915+0000 7fac19a32640 1 librados: init done 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.916+0000 7fac19a32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.916+0000 7fac19a32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.916+0000 7fac19a32640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:06.919 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.916+0000 7fac19a32640 1 -- 172.21.15.69:0/286098054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1413bac0 con 0x7fac140ac700 2023-12-15T17:24:06.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.935+0000 7fac10ff9640 1 -- 172.21.15.69:0/286098054 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fac0c040920 con 0x7fac140ac700 2023-12-15T17:24:06.935 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.935+0000 7fac10ff9640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fabf406cba0 0x7fabf406f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:06.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.935+0000 7fac12ffd640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fabf406cba0 0x7fabf406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:06.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.935+0000 7fac12ffd640 10 monclient: get_auth_request con 0x7fabf406cba0 auth_method 0 2023-12-15T17:24:06.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.936+0000 7fac12ffd640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fabf406cba0 0x7fabf406f060 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fac04004f60 tx=0x7fac04007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:06.936 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:06.936+0000 7fac10ff9640 10 client.21490.objecter ms_handle_connect 0x7fabf406cba0 2023-12-15T17:24:07.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.163+0000 7fac10ff9640 1 -- 172.21.15.69:0/286098054 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7fac0c07b5b0 con 0x7fac140ac700 2023-12-15T17:24:07.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.164+0000 7fac10ff9640 10 client.21490.objecter ms_dispatch 0x7fac14000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:07.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.164+0000 7fac10ff9640 3 client.21490.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:07.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.164+0000 7fac10ff9640 3 client.21490.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:07.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.164+0000 7fac10ff9640 20 client.21490.objecter dump_active .. 0 homeless 2023-12-15T17:24:07.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.176+0000 7fac10ff9640 1 -- 172.21.15.69:0/286098054 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fac0c0778f0 con 0x7fac140ac700 2023-12-15T17:24:07.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.176+0000 7fac10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:07.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.176+0000 7fac10ff9640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:07.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.347+0000 7fac19a32640 1 -- 172.21.15.69:0/286098054 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fac14142be0 con 0x7fabf406cba0 2023-12-15T17:24:07.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.347+0000 7fac10ff9640 1 -- 172.21.15.69:0/286098054 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fac04003940 con 0x7fabf406cba0 2023-12-15T17:24:07.348 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 5 to deep-scrub 2023-12-15T17:24:07.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 10 librados: watch_flush enter 2023-12-15T17:24:07.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 10 librados: watch_flush exit 2023-12-15T17:24:07.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 20 client.21490.objecter shutdown clearing up homeless session... 2023-12-15T17:24:07.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 10 client.21490.objecter successfully canceled tick 2023-12-15T17:24:07.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 -- 172.21.15.69:0/286098054 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fabf406cba0 msgr2=0x7fabf406f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fabf406cba0 0x7fabf406f060 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fac04004f60 tx=0x7fac04007e30 comp rx=0 tx=0).stop 2023-12-15T17:24:07.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 10 monclient: shutdown 2023-12-15T17:24:07.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 -- 172.21.15.69:0/286098054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac140ac700 msgr2=0x7fac1413b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac140ac700 0x7fac1413b3f0 secure :-1 s=READY pgs=5121 cs=0 l=1 rev1=1 crypto rx=0x7fac0c033000 tx=0x7fac0c004a40 comp rx=0 tx=0).stop 2023-12-15T17:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 -- 172.21.15.69:0/286098054 shutdown_connections 2023-12-15T17:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fabf406cba0 0x7fabf406f060 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac140ac700 0x7fac1413b3f0 unknown :-1 s=CLOSED pgs=5121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac1413f370 0x7fac1413aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 --2- 172.21.15.69:0/286098054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac140a8910 0x7fac1413a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 -- 172.21.15.69:0/286098054 >> 172.21.15.69:0/286098054 conn(0x7fac140b7780 msgr2=0x7fac1413bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 -- 172.21.15.69:0/286098054 shutdown_connections 2023-12-15T17:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 -- 172.21.15.69:0/286098054 wait complete. 2023-12-15T17:24:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.350+0000 7fac19a32640 1 librados: shutdown 2023-12-15T17:24:07.366 INFO:tasks.ceph:Scrubbing osd.6 2023-12-15T17:24:07.366 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-15T17:24:07.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.482+0000 7f147b32c640 10 monclient: get_monmap_and_config 2023-12-15T17:24:07.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.482+0000 7f147b32c640 10 monclient: build_initial_monmap 2023-12-15T17:24:07.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.482+0000 7f147b32c640 10 monclient: monmap: 2023-12-15T17:24:07.483 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:07.483 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:07.483 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:07.482888+0000 2023-12-15T17:24:07.483 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:07.482888+0000 2023-12-15T17:24:07.483 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:07.483 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:07.484 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.482+0000 7f147b32c640 1 Processor -- start 2023-12-15T17:24:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 1 -- start start 2023-12-15T17:24:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 10 monclient: init 2023-12-15T17:24:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-12-15T17:24:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1474145840 0x7f1474146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 10 monclient(hunting): picked mon.noname-c con 0x7f1474145840 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f14740b20f0 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14740b2520 addr v1:172.21.15.69:6789/0 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14740aa850 con 0x7f14740b2520 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14740a9260 con 0x7f14740b20f0 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14741465c0 con 0x7f1474145840 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.483+0000 7f147b32c640 10 monclient(hunting): authenticate will time out at 2751.799205s 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.484+0000 7f14790c8640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1474145840 0x7f1474146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.484+0000 7f14790c8640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1474145840 0x7f1474146080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:44042/0 (socket says 172.21.15.69:44042) 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.484+0000 7f14790c8640 1 -- 172.21.15.69:0/2884233150 learned_addr learned my addr 172.21.15.69:0/2884233150 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.484+0000 7f14790c8640 10 monclient(hunting): get_auth_request con 0x7f1474145840 auth_method 0 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.484+0000 7f14790c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.484+0000 7f14790c8640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:07.486 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.484+0000 7f14790c8640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.485+0000 7f14790c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.485+0000 7f14790c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.485+0000 7f14790c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.487+0000 7f14790c8640 10 monclient(hunting): handle_auth_done global_id 21493 payload 274 2023-12-15T17:24:07.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.487+0000 7f14790c8640 1 -- 172.21.15.69:0/2884233150 >> v1:172.21.15.192:6789/0 conn(0x7f14740b20f0 legacy=0x7f147413f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.487+0000 7f14790c8640 1 -- 172.21.15.69:0/2884233150 >> v1:172.21.15.69:6789/0 conn(0x7f14740b2520 legacy=0x7f147414c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:07.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.487+0000 7f14790c8640 10 monclient: _finish_hunting 0 2023-12-15T17:24:07.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.487+0000 7f14790c8640 1 monclient: found mon.noname-c 2023-12-15T17:24:07.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.487+0000 7f14790c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:07.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.487+0000 7f14790c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:24:07.488 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.487+0000 7f14790c8640 1 -- 172.21.15.69:0/2884233150 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147413fc20 con 0x7f1474145840 2023-12-15T17:24:07.491 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.491+0000 7f14790c8640 1 --2- 172.21.15.69:0/2884233150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1474145840 0x7f1474146080 secure :-1 s=READY pgs=5123 cs=0 l=1 rev1=1 crypto rx=0x7f1464009870 tx=0x7f146402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=551964c717ea5469 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:07.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 1 -- 172.21.15.69:0/2884233150 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464031870 con 0x7f1474145840 2023-12-15T17:24:07.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:07.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:07.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 10 monclient: dump: 2023-12-15T17:24:07.495 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:07.495 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:07.495 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.495 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.495 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:07.495 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 10 monclient: _finish_auth 0 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 10 monclient: _check_auth_tickets 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 1 -- 172.21.15.69:0/2884233150 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14640319d0 con 0x7f1474145840 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 1 -- 172.21.15.69:0/2884233150 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464031d60 con 0x7f1474145840 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f146bfff640 10 monclient: dump: 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:07.497 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.497 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.497 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:07.497 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.497 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:07.497 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:07.497 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:07.497 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f147b32c640 5 monclient: authenticate success, global_id 21493 2023-12-15T17:24:07.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f147b32c640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:07.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.494+0000 7f147b32c640 4 set_mon_vals no callback set 2023-12-15T17:24:07.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.495+0000 7f147b32c640 10 monclient: shutdown 2023-12-15T17:24:07.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.495+0000 7f147b32c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:07.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.495+0000 7f147b32c640 1 -- 172.21.15.69:0/2884233150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1474145840 msgr2=0x7f1474146080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.495+0000 7f147b32c640 1 --2- 172.21.15.69:0/2884233150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1474145840 0x7f1474146080 secure :-1 s=READY pgs=5123 cs=0 l=1 rev1=1 crypto rx=0x7f1464009870 tx=0x7f146402b910 comp rx=0 tx=0).stop 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.495+0000 7f147b32c640 1 -- 172.21.15.69:0/2884233150 shutdown_connections 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.495+0000 7f147b32c640 1 --2- 172.21.15.69:0/2884233150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1474145840 0x7f1474146080 unknown :-1 s=CLOSED pgs=5123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.495+0000 7f147b32c640 1 -- 172.21.15.69:0/2884233150 >> 172.21.15.69:0/2884233150 conn(0x7f14740b41f0 msgr2=0x7f14740b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.495+0000 7f147b32c640 1 -- 172.21.15.69:0/2884233150 shutdown_connections 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.495+0000 7f147b32c640 1 -- 172.21.15.69:0/2884233150 wait complete. 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 10 monclient: build_initial_monmap 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 10 monclient: monmap: 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 librados: starting msgr at 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 librados: starting objecter 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 Processor -- start 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 -- start start 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 librados: setting wanted keys 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 librados: calling monclient init 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 10 monclient: init 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14740b20f0 0x7f14741cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 10 monclient(hunting): picked mon.noname-b con 0x7f14740b20f0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14740b2520 0x7f14741cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 10 monclient(hunting): picked mon.noname-a con 0x7f14740b2520 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1474145840 0x7f14741cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 10 monclient(hunting): picked mon.noname-c con 0x7f1474145840 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14741cdde0 con 0x7f14740b2520 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14740b1290 con 0x7f14740b20f0 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.496+0000 7f147b32c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14741cbbc0 con 0x7f1474145840 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.497+0000 7f147b32c640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.497+0000 7f147b32c640 10 monclient(hunting): authenticate will time out at 2751.812370s 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.497+0000 7f14790c8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14740b20f0 0x7f14741cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.497+0000 7f14798c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1474145840 0x7f14741cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.497+0000 7f14790c8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14740b20f0 0x7f14741cb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:44056/0 (socket says 172.21.15.69:44056) 2023-12-15T17:24:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.497+0000 7f14790c8640 1 -- 172.21.15.69:0/2319572163 learned_addr learned my addr 172.21.15.69:0/2319572163 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.497+0000 7f14790c8640 10 monclient(hunting): get_auth_request con 0x7f14740b20f0 auth_method 0 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.497+0000 7f14790c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.497+0000 7f14790c8640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.497+0000 7f14790c8640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.498+0000 7f14798c9640 10 monclient(hunting): get_auth_request con 0x7f1474145840 auth_method 0 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.498+0000 7f14798c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.498+0000 7f14798c9640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.498+0000 7f14798c9640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.498+0000 7f14790c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.498+0000 7f14790c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.498+0000 7f14790c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.498+0000 7f14798c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.498+0000 7f14798c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.498+0000 7f14798c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.499+0000 7f14788c7640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14740b2520 0x7f14741cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.501+0000 7f14790c8640 10 monclient(hunting): handle_auth_done global_id 21496 payload 824 2023-12-15T17:24:07.505 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.501+0000 7f14790c8640 1 -- 172.21.15.69:0/2319572163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1474145840 msgr2=0x7f14741cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.501+0000 7f14790c8640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1474145840 0x7f14741cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.501+0000 7f14790c8640 1 -- 172.21.15.69:0/2319572163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14740b2520 msgr2=0x7f14741cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.501+0000 7f14790c8640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14740b2520 0x7f14741cbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.501+0000 7f14790c8640 10 monclient: _finish_hunting 0 2023-12-15T17:24:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.501+0000 7f14790c8640 1 monclient: found mon.noname-b 2023-12-15T17:24:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.501+0000 7f14790c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.501+0000 7f14790c8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.501+0000 7f14790c8640 1 -- 172.21.15.69:0/2319572163 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640094d0 con 0x7f14740b20f0 2023-12-15T17:24:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.503+0000 7f14790c8640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14740b20f0 0x7f14741cb680 secure :-1 s=READY pgs=5124 cs=0 l=1 rev1=1 crypto rx=0x7f1464009990 tx=0x7f14640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.503+0000 7f1469ffb640 10 client.?.objecter ms_handle_connect 0x7f14740b20f0 2023-12-15T17:24:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.503+0000 7f1469ffb640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:07.507 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 1 -- 172.21.15.69:0/2319572163 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146403adf0 con 0x7f14740b20f0 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 10 monclient: dump: 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 10 monclient: _finish_auth 0 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 10 monclient: _check_auth_tickets 2023-12-15T17:24:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 1 -- 172.21.15.69:0/2319572163 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14640443f0 con 0x7f14740b20f0 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f147b32c640 5 monclient: authenticate success, global_id 21496 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 1 -- 172.21.15.69:0/2319572163 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14640447a0 con 0x7f14740b20f0 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f1469ffb640 10 monclient: dump: 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f146b7fe640 4 set_mon_vals no callback set 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f147b32c640 10 monclient: _renew_subs 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f147b32c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f147b32c640 1 -- 172.21.15.69:0/2319572163 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14741cc940 con 0x7f14740b20f0 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f147b32c640 10 client.21496.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f147b32c640 10 monclient: _renew_subs 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f147b32c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f147b32c640 1 -- 172.21.15.69:0/2319572163 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14741cd6e0 con 0x7f14740b20f0 2023-12-15T17:24:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.506+0000 7f147b32c640 1 librados: init done 2023-12-15T17:24:07.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.507+0000 7f147b32c640 10 client.21496.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:07.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.507+0000 7f147b32c640 15 client.21496.objecter _session_command_op_assign -1 1 2023-12-15T17:24:07.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.507+0000 7f147b32c640 20 client.21496.objecter _get_session osd=-1 returning homeless 2023-12-15T17:24:07.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.507+0000 7f147b32c640 10 client.21496.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:07.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.507+0000 7f147b32c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:07.511 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.507+0000 7f147b32c640 1 -- 172.21.15.69:0/2319572163 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14740aad80 con 0x7f14740b20f0 2023-12-15T17:24:07.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.525+0000 7f1469ffb640 1 -- 172.21.15.69:0/2319572163 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f146404d450 con 0x7f14740b20f0 2023-12-15T17:24:07.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.525+0000 7f1469ffb640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f145806cb50 0x7f145806f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.525+0000 7f14788c7640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f145806cb50 0x7f145806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.526 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.526+0000 7f14788c7640 10 monclient: get_auth_request con 0x7f145806cb50 auth_method 0 2023-12-15T17:24:07.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.526+0000 7f1469ffb640 10 client.21496.objecter ms_handle_connect 0x7f145806cb50 2023-12-15T17:24:07.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.526+0000 7f14788c7640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f145806cb50 0x7f145806f010 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1474141c00 tx=0x7f1470009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:07.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 1 -- 172.21.15.69:0/2319572163 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7f146402d040 con 0x7f14740b20f0 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 10 client.21496.objecter ms_dispatch 0x7f1474000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 3 client.21496.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 3 client.21496.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 10 client.21496.objecter checking command 1 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] conn(0x7f14580743e0 0x7f1458076880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 20 client.21496.objecter _get_session s=0x7f1458073d10 osd=6 3 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 20 client.21496.objecter put_session s=0x7f1458073d10 osd=6 3 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 15 client.21496.objecter _session_command_op_remove -1 1 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 20 client.21496.objecter _get_session s=0x7f1458073d10 osd=6 3 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 20 client.21496.objecter get_session s=0x7f1458073d10 osd=6 3 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 15 client.21496.objecter _session_command_op_assign 6 1 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 20 client.21496.objecter put_session s=0x7f1458073d10 osd=6 4 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 10 client.21496.objecter _send_command 1 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 1 -- 172.21.15.69:0/2319572163 --> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1458076f50 con 0x7f14580743e0 2023-12-15T17:24:07.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.527+0000 7f1469ffb640 20 client.21496.objecter dump_active .. 0 homeless 2023-12-15T17:24:07.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.528+0000 7f14798c9640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] conn(0x7f14580743e0 0x7f1458076880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.528+0000 7f14798c9640 10 monclient: get_auth_request con 0x7f14580743e0 auth_method 0 2023-12-15T17:24:07.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.528+0000 7f1469ffb640 1 -- 172.21.15.69:0/2319572163 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=908) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1464077360 con 0x7f14740b20f0 2023-12-15T17:24:07.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.528+0000 7f1469ffb640 10 monclient: handle_get_version_reply finishing 1 version 908 2023-12-15T17:24:07.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.531+0000 7f1469ffb640 10 client.21496.objecter ms_handle_connect 0x7f14580743e0 2023-12-15T17:24:07.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.531+0000 7f14798c9640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] conn(0x7f14580743e0 0x7f1458076880 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:07.555 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.555+0000 7f1469ffb640 1 -- 172.21.15.69:0/2319572163 <== osd.6 v2:172.21.15.192:6818/2563835645 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1458076f50 con 0x7f14580743e0 2023-12-15T17:24:07.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.555+0000 7f1469ffb640 10 client.21496.objecter ms_dispatch 0x7f1474000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:24:07.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.555+0000 7f1469ffb640 10 client.21496.objecter _finish_command 1 = system:0 2023-12-15T17:24:07.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.555+0000 7f1469ffb640 20 client.21496.objecter put_session s=0x7f1458073d10 osd=6 4 2023-12-15T17:24:07.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.555+0000 7f1469ffb640 15 client.21496.objecter _session_command_op_remove 6 1 2023-12-15T17:24:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.567+0000 7f147b32c640 10 client.21496.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-15T17:24:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.567+0000 7f147b32c640 15 client.21496.objecter _session_command_op_assign -1 2 2023-12-15T17:24:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.567+0000 7f147b32c640 20 client.21496.objecter _get_session s=0x7f1458073d10 osd=6 3 2023-12-15T17:24:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.567+0000 7f147b32c640 20 client.21496.objecter put_session s=0x7f1458073d10 osd=6 3 2023-12-15T17:24:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.567+0000 7f147b32c640 20 client.21496.objecter _get_session s=0x7f1458073d10 osd=6 3 2023-12-15T17:24:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.567+0000 7f147b32c640 15 client.21496.objecter _session_command_op_remove -1 2 2023-12-15T17:24:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.567+0000 7f147b32c640 20 client.21496.objecter get_session s=0x7f1458073d10 osd=6 3 2023-12-15T17:24:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.567+0000 7f147b32c640 15 client.21496.objecter _session_command_op_assign 6 2 2023-12-15T17:24:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.567+0000 7f147b32c640 20 client.21496.objecter put_session s=0x7f1458073d10 osd=6 4 2023-12-15T17:24:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.567+0000 7f147b32c640 10 client.21496.objecter _send_command 2 2023-12-15T17:24:07.568 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.567+0000 7f147b32c640 1 -- 172.21.15.69:0/2319572163 --> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1474145c20 con 0x7f14580743e0 2023-12-15T17:24:07.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f1469ffb640 1 -- 172.21.15.69:0/2319572163 <== osd.6 v2:172.21.15.192:6818/2563835645 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f146c001ac0 con 0x7f14580743e0 2023-12-15T17:24:07.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f1469ffb640 10 client.21496.objecter ms_dispatch 0x7f1474000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:24:07.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f1469ffb640 10 client.21496.objecter _finish_command 2 = system:0 2023-12-15T17:24:07.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f1469ffb640 20 client.21496.objecter put_session s=0x7f1458073d10 osd=6 4 2023-12-15T17:24:07.583 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f1469ffb640 15 client.21496.objecter _session_command_op_remove 6 2 2023-12-15T17:24:07.583 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-12-15T17:24:07.583 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-15T17:24:07.583 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-12-15T17:24:07.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 10 librados: watch_flush enter 2023-12-15T17:24:07.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 10 librados: watch_flush exit 2023-12-15T17:24:07.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 10 client.21496.objecter close_session for osd.6 2023-12-15T17:24:07.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 1 -- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] conn(0x7f14580743e0 msgr2=0x7f1458076880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:6818/2563835645,v1:172.21.15.192:6819/2563835645] conn(0x7f14580743e0 0x7f1458076880 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 20 client.21496.objecter put_session s=0x7f1458073d10 osd=6 1 2023-12-15T17:24:07.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 20 client.21496.objecter shutdown clearing up homeless session... 2023-12-15T17:24:07.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 10 client.21496.objecter successfully canceled tick 2023-12-15T17:24:07.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 1 -- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f145806cb50 msgr2=0x7f145806f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f145806cb50 0x7f145806f010 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1474141c00 tx=0x7f1470009040 comp rx=0 tx=0).stop 2023-12-15T17:24:07.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 10 monclient: shutdown 2023-12-15T17:24:07.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:07.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 1 -- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14740b20f0 msgr2=0x7f14741cb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 1 --2- 172.21.15.69:0/2319572163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14740b20f0 0x7f14741cb680 secure :-1 s=READY pgs=5124 cs=0 l=1 rev1=1 crypto rx=0x7f1464009990 tx=0x7f14640046c0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f14798c9640 1 -- 172.21.15.69:0/2319572163 reap_dead start 2023-12-15T17:24:07.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 1 -- 172.21.15.69:0/2319572163 shutdown_connections 2023-12-15T17:24:07.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 1 -- 172.21.15.69:0/2319572163 >> 172.21.15.69:0/2319572163 conn(0x7f14740b41f0 msgr2=0x7f14740b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:07.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 1 -- 172.21.15.69:0/2319572163 shutdown_connections 2023-12-15T17:24:07.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.582+0000 7f147b32c640 1 -- 172.21.15.69:0/2319572163 wait complete. 2023-12-15T17:24:07.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.583+0000 7f147b32c640 1 librados: shutdown 2023-12-15T17:24:07.595 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-15T17:24:07.704 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.704+0000 7fa56e0ec640 10 monclient: get_monmap_and_config 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.704+0000 7fa56e0ec640 10 monclient: build_initial_monmap 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.704+0000 7fa56e0ec640 10 monclient: monmap: 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:07.704974+0000 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:07.704974+0000 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:07.705 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:07.706 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:07.706 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 1 Processor -- start 2023-12-15T17:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 1 -- start start 2023-12-15T17:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 10 monclient: init 2023-12-15T17:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-12-15T17:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa5680a8a30 0x7fa5681424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 10 monclient(hunting): picked mon.noname-c con 0x7fa5680a8a30 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:07.708 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5680a8e10 0x7fa568142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 10 monclient(hunting): picked mon.noname-b con 0x7fa5680a8e10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5681454a0 0x7fa568145930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 10 monclient(hunting): picked mon.noname-a con 0x7fa5681454a0 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5680aaa10 con 0x7fa5681454a0 2023-12-15T17:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5680aac10 con 0x7fa5680a8a30 2023-12-15T17:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680aaef0 con 0x7fa5680a8e10 2023-12-15T17:24:07.709 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.705+0000 7fa56e0ec640 10 monclient(hunting): authenticate will time out at 2752.021342s 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.706+0000 7fa567fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5681454a0 0x7fa568145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.706+0000 7fa566ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5680a8e10 0x7fa568142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.706+0000 7fa5677fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa5680a8a30 0x7fa5681424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.706+0000 7fa567fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5681454a0 0x7fa568145930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45508/0 (socket says 172.21.15.69:45508) 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.706+0000 7fa567fff640 1 -- 172.21.15.69:0/4149965492 learned_addr learned my addr 172.21.15.69:0/4149965492 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.706+0000 7fa567fff640 10 monclient(hunting): get_auth_request con 0x7fa5681454a0 auth_method 0 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.706+0000 7fa567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.706+0000 7fa567fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.706+0000 7fa567fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa566ffd640 10 monclient(hunting): get_auth_request con 0x7fa5680a8e10 auth_method 0 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa566ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa566ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:07.710 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa566ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa5677fe640 10 monclient(hunting): get_auth_request con 0x7fa5680a8a30 auth_method 0 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa5677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa5677fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa5677fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa566ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa566ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa566ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa5677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa5677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:07.711 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.707+0000 7fa5677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:07.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.709+0000 7fa5677fe640 10 monclient(hunting): handle_auth_done global_id 21499 payload 274 2023-12-15T17:24:07.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.709+0000 7fa5677fe640 1 -- 172.21.15.69:0/4149965492 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5680a8e10 msgr2=0x7fa568142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:07.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.709+0000 7fa5677fe640 1 --2- 172.21.15.69:0/4149965492 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5680a8e10 0x7fa568142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.709+0000 7fa5677fe640 1 -- 172.21.15.69:0/4149965492 >> v2:172.21.15.69:3300/0 conn(0x7fa5681454a0 msgr2=0x7fa568145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:07.712 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.709+0000 7fa5677fe640 1 --2- 172.21.15.69:0/4149965492 >> v2:172.21.15.69:3300/0 conn(0x7fa5681454a0 0x7fa568145930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.709+0000 7fa5677fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:07.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.709+0000 7fa5677fe640 1 monclient: found mon.noname-c 2023-12-15T17:24:07.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.709+0000 7fa5677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:07.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.709+0000 7fa5677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.192:3300/0 2023-12-15T17:24:07.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.709+0000 7fa5677fe640 1 -- 172.21.15.69:0/4149965492 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5680a97f0 con 0x7fa5680a8a30 2023-12-15T17:24:07.713 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.711+0000 7fa5677fe640 1 --2- 172.21.15.69:0/4149965492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5680a8a30 0x7fa5681424e0 secure :-1 s=READY pgs=5125 cs=0 l=1 rev1=1 crypto rx=0x7fa5580097a0 tx=0x7fa55802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a295e05fb1266e11 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:07.714 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 1 -- 172.21.15.69:0/4149965492 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55802e070 con 0x7fa5680a8a30 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 10 monclient: dump: 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:07.715 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 10 monclient: _finish_auth 0 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 10 monclient: _check_auth_tickets 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 1 -- 172.21.15.69:0/4149965492 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5580315a0 con 0x7fa5680a8a30 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 1 -- 172.21.15.69:0/4149965492 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558031930 con 0x7fa5680a8a30 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa5667fc640 10 monclient: dump: 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:07.716 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa56e0ec640 5 monclient: authenticate success, global_id 21499 2023-12-15T17:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa56e0ec640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:07.717 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.714+0000 7fa56e0ec640 4 set_mon_vals no callback set 2023-12-15T17:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.715+0000 7fa56e0ec640 10 monclient: shutdown 2023-12-15T17:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.715+0000 7fa56e0ec640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.715+0000 7fa56e0ec640 1 -- 172.21.15.69:0/4149965492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5680a8a30 msgr2=0x7fa5681424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.715+0000 7fa56e0ec640 1 --2- 172.21.15.69:0/4149965492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5680a8a30 0x7fa5681424e0 secure :-1 s=READY pgs=5125 cs=0 l=1 rev1=1 crypto rx=0x7fa5580097a0 tx=0x7fa55802b840 comp rx=0 tx=0).stop 2023-12-15T17:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.715+0000 7fa56e0ec640 1 -- 172.21.15.69:0/4149965492 shutdown_connections 2023-12-15T17:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.715+0000 7fa56e0ec640 1 --2- 172.21.15.69:0/4149965492 >> v2:172.21.15.69:3300/0 conn(0x7fa5681454a0 0x7fa568145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.715+0000 7fa56e0ec640 1 --2- 172.21.15.69:0/4149965492 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5680a8e10 0x7fa568142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.715+0000 7fa56e0ec640 1 --2- 172.21.15.69:0/4149965492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5680a8a30 0x7fa5681424e0 unknown :-1 s=CLOSED pgs=5125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.715+0000 7fa56e0ec640 1 -- 172.21.15.69:0/4149965492 >> 172.21.15.69:0/4149965492 conn(0x7fa568061a30 msgr2=0x7fa5680b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.715+0000 7fa56e0ec640 1 -- 172.21.15.69:0/4149965492 shutdown_connections 2023-12-15T17:24:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.715+0000 7fa56e0ec640 1 -- 172.21.15.69:0/4149965492 wait complete. 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.716+0000 7fa56e0ec640 10 monclient: build_initial_monmap 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.716+0000 7fa56e0ec640 10 monclient: monmap: 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.716+0000 7fa56e0ec640 1 librados: starting msgr at 2023-12-15T17:24:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.716+0000 7fa56e0ec640 1 librados: starting objecter 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.716+0000 7fa56e0ec640 1 Processor -- start 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.716+0000 7fa56e0ec640 1 -- start start 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.716+0000 7fa56e0ec640 1 librados: setting wanted keys 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.716+0000 7fa56e0ec640 1 librados: calling monclient init 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.716+0000 7fa56e0ec640 10 monclient: init 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5680a8a30 0x7fa5681da5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 10 monclient(hunting): picked mon.noname-c con 0x7fa5680a8a30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5680a8e10 0x7fa56813b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 10 monclient(hunting): picked mon.noname-b con 0x7fa5680a8e10 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5681454a0 0x7fa56813bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 10 monclient(hunting): picked mon.noname-a con 0x7fa5681454a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680adff0 con 0x7fa5681454a0 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680aec30 con 0x7fa5680a8e10 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56813c0f0 con 0x7fa5680a8a30 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa56e0ec640 10 monclient(hunting): authenticate will time out at 2752.032717s 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa5677fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5680a8a30 0x7fa5681da5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.717+0000 7fa567fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5681454a0 0x7fa56813bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa5677fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5680a8a30 0x7fa5681da5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42514/0 (socket says 172.21.15.69:42514) 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa5677fe640 1 -- 172.21.15.69:0/3465320754 learned_addr learned my addr 172.21.15.69:0/3465320754 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa5677fe640 10 monclient(hunting): get_auth_request con 0x7fa5680a8a30 auth_method 0 2023-12-15T17:24:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa5677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa5677fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa5677fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa567fff640 10 monclient(hunting): get_auth_request con 0x7fa5681454a0 auth_method 0 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa567fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa567fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa566ffd640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5680a8e10 0x7fa56813b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa566ffd640 10 monclient(hunting): get_auth_request con 0x7fa5680a8e10 auth_method 0 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa566ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa566ffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa566ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa5677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa5677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa5677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.718+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.719+0000 7fa566ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.719+0000 7fa566ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.719+0000 7fa566ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:07.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.721+0000 7fa566ffd640 10 monclient(hunting): handle_auth_done global_id 21502 payload 824 2023-12-15T17:24:07.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.721+0000 7fa566ffd640 1 -- 172.21.15.69:0/3465320754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5680a8a30 msgr2=0x7fa5681da5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.721+0000 7fa566ffd640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5680a8a30 0x7fa5681da5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.721+0000 7fa566ffd640 1 -- 172.21.15.69:0/3465320754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5681454a0 msgr2=0x7fa56813bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.721+0000 7fa566ffd640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5681454a0 0x7fa56813bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.721+0000 7fa566ffd640 10 monclient: _finish_hunting 0 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.721+0000 7fa566ffd640 1 monclient: found mon.noname-b 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.721+0000 7fa566ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.721+0000 7fa566ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.721+0000 7fa566ffd640 1 -- 172.21.15.69:0/3465320754 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa558009400 con 0x7fa5680a8e10 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.723+0000 7fa566ffd640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5680a8e10 0x7fa56813b670 secure :-1 s=READY pgs=5126 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00e8a0 tx=0x7fa55c00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.723+0000 7fa54ffff640 10 client.?.objecter ms_handle_connect 0x7fa5680a8e10 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.723+0000 7fa54ffff640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 1 -- 172.21.15.69:0/3465320754 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c00c940 con 0x7fa5680a8e10 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 10 monclient: dump: 2023-12-15T17:24:07.728 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 10 monclient: _finish_auth 0 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 10 monclient: _check_auth_tickets 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 1 -- 172.21.15.69:0/3465320754 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55c00caa0 con 0x7fa5680a8e10 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:07.729 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 1 -- 172.21.15.69:0/3465320754 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c00ce10 con 0x7fa5680a8e10 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa56e0ec640 5 monclient: authenticate success, global_id 21502 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa56e0ec640 10 monclient: _renew_subs 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa56e0ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa56e0ec640 1 -- 172.21.15.69:0/3465320754 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56813c3c0 con 0x7fa5680a8e10 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa54ffff640 10 monclient: dump: 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa565ffb640 4 set_mon_vals no callback set 2023-12-15T17:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa56e0ec640 10 client.21502.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa56e0ec640 10 monclient: _renew_subs 2023-12-15T17:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa56e0ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa56e0ec640 1 -- 172.21.15.69:0/3465320754 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5680b1740 con 0x7fa5680a8e10 2023-12-15T17:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.727+0000 7fa56e0ec640 1 librados: init done 2023-12-15T17:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.728+0000 7fa56e0ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.728+0000 7fa56e0ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:07.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.728+0000 7fa56e0ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:07.732 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.728+0000 7fa56e0ec640 1 -- 172.21.15.69:0/3465320754 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa568067ac0 con 0x7fa5680a8e10 2023-12-15T17:24:07.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.745+0000 7fa54ffff640 1 -- 172.21.15.69:0/3465320754 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa55c013470 con 0x7fa5680a8e10 2023-12-15T17:24:07.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.746+0000 7fa54ffff640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55406cb50 0x7fa55406f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:07.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.746+0000 7fa5677fe640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55406cb50 0x7fa55406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:07.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.746+0000 7fa5677fe640 10 monclient: get_auth_request con 0x7fa55406cb50 auth_method 0 2023-12-15T17:24:07.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.746+0000 7fa54ffff640 10 client.21502.objecter ms_handle_connect 0x7fa55406cb50 2023-12-15T17:24:07.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.746+0000 7fa5677fe640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55406cb50 0x7fa55406f010 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa55802b880 tx=0x7fa55803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:07.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.747+0000 7fa54ffff640 1 -- 172.21.15.69:0/3465320754 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7fa55c058d50 con 0x7fa5680a8e10 2023-12-15T17:24:07.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.747+0000 7fa54ffff640 10 client.21502.objecter ms_dispatch 0x7fa568000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:07.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.747+0000 7fa54ffff640 3 client.21502.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:07.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.747+0000 7fa54ffff640 3 client.21502.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:07.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.747+0000 7fa54ffff640 20 client.21502.objecter dump_active .. 0 homeless 2023-12-15T17:24:07.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.982+0000 7fa54ffff640 1 -- 172.21.15.69:0/3465320754 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa55c055090 con 0x7fa5680a8e10 2023-12-15T17:24:07.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.982+0000 7fa54ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:07.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:07.982+0000 7fa54ffff640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:08.160 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.160+0000 7fa56e0ec640 1 -- 172.21.15.69:0/3465320754 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5680b3520 con 0x7fa55406cb50 2023-12-15T17:24:08.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.161+0000 7fa54ffff640 1 -- 172.21.15.69:0/3465320754 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa5680b3520 con 0x7fa55406cb50 2023-12-15T17:24:08.161 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 6 to deep-scrub 2023-12-15T17:24:08.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 10 librados: watch_flush enter 2023-12-15T17:24:08.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 10 librados: watch_flush exit 2023-12-15T17:24:08.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 20 client.21502.objecter shutdown clearing up homeless session... 2023-12-15T17:24:08.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 10 client.21502.objecter successfully canceled tick 2023-12-15T17:24:08.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 -- 172.21.15.69:0/3465320754 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55406cb50 msgr2=0x7fa55406f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55406cb50 0x7fa55406f010 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa55802b880 tx=0x7fa55803d040 comp rx=0 tx=0).stop 2023-12-15T17:24:08.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 10 monclient: shutdown 2023-12-15T17:24:08.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:08.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 -- 172.21.15.69:0/3465320754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5680a8e10 msgr2=0x7fa56813b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5680a8e10 0x7fa56813b670 secure :-1 s=READY pgs=5126 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00e8a0 tx=0x7fa55c00ed60 comp rx=0 tx=0).stop 2023-12-15T17:24:08.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 -- 172.21.15.69:0/3465320754 shutdown_connections 2023-12-15T17:24:08.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fa55406cb50 0x7fa55406f010 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5681454a0 0x7fa56813bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5680a8e10 0x7fa56813b670 unknown :-1 s=CLOSED pgs=5126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 --2- 172.21.15.69:0/3465320754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5680a8a30 0x7fa5681da5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 -- 172.21.15.69:0/3465320754 >> 172.21.15.69:0/3465320754 conn(0x7fa568061a30 msgr2=0x7fa56813eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:08.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 -- 172.21.15.69:0/3465320754 shutdown_connections 2023-12-15T17:24:08.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 -- 172.21.15.69:0/3465320754 wait complete. 2023-12-15T17:24:08.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.163+0000 7fa56e0ec640 1 librados: shutdown 2023-12-15T17:24:08.174 INFO:tasks.ceph:Scrubbing osd.7 2023-12-15T17:24:08.174 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-15T17:24:08.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.283+0000 7fd34f832640 10 monclient: get_monmap_and_config 2023-12-15T17:24:08.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.283+0000 7fd34f832640 10 monclient: build_initial_monmap 2023-12-15T17:24:08.284 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.284+0000 7fd34f832640 10 monclient: monmap: 2023-12-15T17:24:08.284 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:08.284 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:08.284 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:08.284458+0000 2023-12-15T17:24:08.284 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:08.284458+0000 2023-12-15T17:24:08.285 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:08.285 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.285 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:08.285 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:08.285 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:08.285 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:08.285 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:08.285 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.284+0000 7fd34f832640 1 Processor -- start 2023-12-15T17:24:08.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.284+0000 7fd34f832640 1 -- start start 2023-12-15T17:24:08.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.284+0000 7fd34f832640 10 monclient: init 2023-12-15T17:24:08.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:08.286 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3480a88a0 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd34813f7c0 0x7fd3480a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 10 monclient(hunting): picked mon.noname-c con 0x7fd34813f7c0 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd348145480 0x7fd348145910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 10 monclient(hunting): picked mon.noname-b con 0x7fd348145480 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3480b17d0 con 0x7fd3480a88a0 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3480aac60 con 0x7fd34813f7c0 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480b1560 con 0x7fd348145480 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:08.287 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.285+0000 7fd34f832640 10 monclient(hunting): authenticate will time out at 2752.600836s 2023-12-15T17:24:08.297 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.297+0000 7fd34ddcf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd348145480 0x7fd348145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:08.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.297+0000 7fd34ddcf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd348145480 0x7fd348145910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42526/0 (socket says 172.21.15.69:42526) 2023-12-15T17:24:08.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.297+0000 7fd34ddcf640 1 -- 172.21.15.69:0/1151565843 learned_addr learned my addr 172.21.15.69:0/1151565843 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:08.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.297+0000 7fd34ddcf640 10 monclient(hunting): get_auth_request con 0x7fd348145480 auth_method 0 2023-12-15T17:24:08.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.297+0000 7fd34ddcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:08.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.297+0000 7fd34ddcf640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:08.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.297+0000 7fd34ddcf640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.298+0000 7fd34cdcd640 1 --2- 172.21.15.69:0/1151565843 >> v2:172.21.15.192:3300/0 conn(0x7fd34813f7c0 0x7fd3480a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.298+0000 7fd34ddcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.298+0000 7fd34ddcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.298+0000 7fd34ddcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.298+0000 7fd34cdcd640 10 monclient(hunting): get_auth_request con 0x7fd34813f7c0 auth_method 0 2023-12-15T17:24:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.298+0000 7fd34cdcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.298+0000 7fd34cdcd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.298+0000 7fd34cdcd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34cdcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34cdcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34cdcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34ddcf640 10 monclient(hunting): handle_auth_done global_id 17846 payload 274 2023-12-15T17:24:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34ddcf640 1 -- 172.21.15.69:0/1151565843 >> v2:172.21.15.192:3300/0 conn(0x7fd34813f7c0 msgr2=0x7fd3480a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34ddcf640 1 --2- 172.21.15.69:0/1151565843 >> v2:172.21.15.192:3300/0 conn(0x7fd34813f7c0 0x7fd3480a8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34ddcf640 1 -- 172.21.15.69:0/1151565843 >> v1:172.21.15.192:6789/0 conn(0x7fd3480a88a0 legacy=0x7fd34813f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34ddcf640 10 monclient: _finish_hunting 0 2023-12-15T17:24:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34ddcf640 1 monclient: found mon.noname-b 2023-12-15T17:24:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34ddcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34ddcf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:24:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.300+0000 7fd34ddcf640 1 -- 172.21.15.69:0/1151565843 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480a97d0 con 0x7fd348145480 2023-12-15T17:24:08.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.302+0000 7fd34ddcf640 1 --2- 172.21.15.69:0/1151565843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd348145480 0x7fd348145910 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7fd340009e50 tx=0x7fd34002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91609bd1f5abe15e server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:08.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.304+0000 7fd33ffff640 1 -- 172.21.15.69:0/1151565843 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34002ca70 con 0x7fd348145480 2023-12-15T17:24:08.304 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.304+0000 7fd33ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.304+0000 7fd33ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.304+0000 7fd33ffff640 10 monclient: dump: 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.304+0000 7fd33ffff640 10 monclient: _finish_auth 0 2023-12-15T17:24:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.304+0000 7fd33ffff640 10 monclient: _check_auth_tickets 2023-12-15T17:24:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.304+0000 7fd33ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.304+0000 7fd34f832640 5 monclient: authenticate success, global_id 17846 2023-12-15T17:24:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.304+0000 7fd34f832640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:24:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.306+0000 7fd33ffff640 1 -- 172.21.15.69:0/1151565843 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34002cbd0 con 0x7fd348145480 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.306+0000 7fd33ffff640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.306+0000 7fd33ffff640 1 -- 172.21.15.69:0/1151565843 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3400343f0 con 0x7fd348145480 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.306+0000 7fd34f832640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.306+0000 7fd34f832640 4 set_mon_vals no callback set 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.306+0000 7fd33ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.306+0000 7fd33ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.306+0000 7fd33ffff640 10 monclient: dump: 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.308 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:08.308 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:08.308 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:08.308 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.310 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 10 monclient: shutdown 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 1 -- 172.21.15.69:0/1151565843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd348145480 msgr2=0x7fd348145910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 1 --2- 172.21.15.69:0/1151565843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd348145480 0x7fd348145910 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7fd340009e50 tx=0x7fd34002bed0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 1 -- 172.21.15.69:0/1151565843 shutdown_connections 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 1 --2- 172.21.15.69:0/1151565843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd348145480 0x7fd348145910 unknown :-1 s=CLOSED pgs=3136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 1 --2- 172.21.15.69:0/1151565843 >> v2:172.21.15.192:3300/0 conn(0x7fd34813f7c0 0x7fd3480a8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 1 -- 172.21.15.69:0/1151565843 >> 172.21.15.69:0/1151565843 conn(0x7fd3480619a0 msgr2=0x7fd3480b7780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 1 -- 172.21.15.69:0/1151565843 shutdown_connections 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 1 -- 172.21.15.69:0/1151565843 wait complete. 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 10 monclient: build_initial_monmap 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.307+0000 7fd34f832640 10 monclient: monmap: 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:08.311 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 librados: starting msgr at 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 librados: starting objecter 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 Processor -- start 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 -- start start 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 librados: setting wanted keys 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 librados: calling monclient init 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 10 monclient: init 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3480a88a0 0x7fd348139f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 10 monclient(hunting): picked mon.noname-a con 0x7fd3480a88a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34813f7c0 0x7fd3481347e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 10 monclient(hunting): picked mon.noname-c con 0x7fd34813f7c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd348145480 0x7fd348136130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 10 monclient(hunting): picked mon.noname-b con 0x7fd348145480 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd348136b40 con 0x7fd3480a88a0 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34813a990 con 0x7fd348145480 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34813ab00 con 0x7fd34813f7c0 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.308+0000 7fd34f832640 10 monclient(hunting): authenticate will time out at 2752.624159s 2023-12-15T17:24:08.313 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.309+0000 7fd34cdcd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34813f7c0 0x7fd3481347e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.309+0000 7fd34cdcd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34813f7c0 0x7fd3481347e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42540/0 (socket says 172.21.15.69:42540) 2023-12-15T17:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.309+0000 7fd34cdcd640 1 -- 172.21.15.69:0/60907252 learned_addr learned my addr 172.21.15.69:0/60907252 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.309+0000 7fd34cdcd640 10 monclient(hunting): get_auth_request con 0x7fd34813f7c0 auth_method 0 2023-12-15T17:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.309+0000 7fd34cdcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.309+0000 7fd34cdcd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.309+0000 7fd34cdcd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.310+0000 7fd34cdcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.310+0000 7fd34cdcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.310+0000 7fd34cdcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:08.314 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.312+0000 7fd34cdcd640 10 monclient(hunting): handle_auth_done global_id 17849 payload 824 2023-12-15T17:24:08.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.312+0000 7fd34cdcd640 1 -- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd348145480 msgr2=0x7fd348136130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.312+0000 7fd34cdcd640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd348145480 0x7fd348136130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.312+0000 7fd34cdcd640 1 -- 172.21.15.69:0/60907252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3480a88a0 msgr2=0x7fd348139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.312+0000 7fd34cdcd640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3480a88a0 0x7fd348139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.312+0000 7fd34cdcd640 10 monclient: _finish_hunting 0 2023-12-15T17:24:08.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.312+0000 7fd34cdcd640 1 monclient: found mon.noname-c 2023-12-15T17:24:08.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.312+0000 7fd34cdcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:08.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.312+0000 7fd34cdcd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:24:08.315 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.312+0000 7fd34cdcd640 1 -- 172.21.15.69:0/60907252 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340009b30 con 0x7fd34813f7c0 2023-12-15T17:24:08.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.315+0000 7fd34cdcd640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34813f7c0 0x7fd3481347e0 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7fd34400e8d0 tx=0x7fd34400ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:08.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.315+0000 7fd33dffb640 10 client.?.objecter ms_handle_connect 0x7fd34813f7c0 2023-12-15T17:24:08.316 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.315+0000 7fd33dffb640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:08.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd33dffb640 1 -- 172.21.15.69:0/60907252 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd344009650 con 0x7fd34813f7c0 2023-12-15T17:24:08.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd33dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:08.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd33dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:08.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd33dffb640 10 monclient: dump: 2023-12-15T17:24:08.377 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:08.377 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:08.377 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.377 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd33dffb640 10 monclient: _finish_auth 0 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd33dffb640 10 monclient: _check_auth_tickets 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd33dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd34f832640 5 monclient: authenticate success, global_id 17849 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd34f832640 10 monclient: _renew_subs 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd34f832640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd34f832640 1 -- 172.21.15.69:0/60907252 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd348136670 con 0x7fd34813f7c0 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd34f832640 10 client.17849.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd34f832640 10 monclient: _renew_subs 2023-12-15T17:24:08.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd34f832640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:08.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd34f832640 1 -- 172.21.15.69:0/60907252 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3481371b0 con 0x7fd34813f7c0 2023-12-15T17:24:08.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.376+0000 7fd34f832640 1 librados: init done 2023-12-15T17:24:08.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.377+0000 7fd34f832640 10 client.17849.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:08.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.377+0000 7fd34f832640 15 client.17849.objecter _session_command_op_assign -1 1 2023-12-15T17:24:08.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.377+0000 7fd34f832640 20 client.17849.objecter _get_session osd=-1 returning homeless 2023-12-15T17:24:08.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.377+0000 7fd34f832640 10 client.17849.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:08.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.377+0000 7fd34f832640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:08.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.377+0000 7fd34f832640 1 -- 172.21.15.69:0/60907252 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd3480678f0 con 0x7fd34813f7c0 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.403+0000 7fd33dffb640 1 -- 172.21.15.69:0/60907252 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3480678f0 con 0x7fd34813f7c0 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.403+0000 7fd33dffb640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.403+0000 7fd33dffb640 1 -- 172.21.15.69:0/60907252 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd344004440 con 0x7fd34813f7c0 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.403+0000 7fd33dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.403+0000 7fd33dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.403+0000 7fd33dffb640 10 monclient: dump: 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.404 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:08.405 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:08.405 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:08.405 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.405 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.403+0000 7fd33f7fe640 4 set_mon_vals no callback set 2023-12-15T17:24:08.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.404+0000 7fd33dffb640 1 -- 172.21.15.69:0/60907252 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd344005990 con 0x7fd34813f7c0 2023-12-15T17:24:08.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.404+0000 7fd33dffb640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fd32c06cb80 0x7fd32c06f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.404+0000 7fd34d5ce640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fd32c06cb80 0x7fd32c06f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:08.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.404+0000 7fd34d5ce640 10 monclient: get_auth_request con 0x7fd32c06cb80 auth_method 0 2023-12-15T17:24:08.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.405+0000 7fd34d5ce640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fd32c06cb80 0x7fd32c06f040 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd338000f60 tx=0x7fd3380079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:08.406 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.405+0000 7fd33dffb640 10 client.17849.objecter ms_handle_connect 0x7fd32c06cb80 2023-12-15T17:24:08.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 1 -- 172.21.15.69:0/60907252 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7fd3440592f0 con 0x7fd34813f7c0 2023-12-15T17:24:08.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 10 client.17849.objecter ms_dispatch 0x7fd348000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:08.409 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 3 client.17849.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 3 client.17849.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 10 client.17849.objecter checking command 1 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] conn(0x7fd32c074410 0x7fd32c0768b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 20 client.17849.objecter _get_session s=0x7fd32c073d40 osd=7 3 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 20 client.17849.objecter put_session s=0x7fd32c073d40 osd=7 3 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 15 client.17849.objecter _session_command_op_remove -1 1 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 20 client.17849.objecter _get_session s=0x7fd32c073d40 osd=7 3 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 20 client.17849.objecter get_session s=0x7fd32c073d40 osd=7 3 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 15 client.17849.objecter _session_command_op_assign 7 1 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 20 client.17849.objecter put_session s=0x7fd32c073d40 osd=7 4 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 10 client.17849.objecter _send_command 1 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 1 -- 172.21.15.69:0/60907252 --> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd32c076f80 con 0x7fd32c074410 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 20 client.17849.objecter dump_active .. 0 homeless 2023-12-15T17:24:08.410 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 1 -- 172.21.15.69:0/60907252 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=908) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3440589c0 con 0x7fd34813f7c0 2023-12-15T17:24:08.411 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.408+0000 7fd33dffb640 10 monclient: handle_get_version_reply finishing 1 version 908 2023-12-15T17:24:08.412 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.412+0000 7fd34ddcf640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] conn(0x7fd32c074410 0x7fd32c0768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:08.413 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.412+0000 7fd34ddcf640 10 monclient: get_auth_request con 0x7fd32c074410 auth_method 0 2023-12-15T17:24:08.415 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.415+0000 7fd34ddcf640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] conn(0x7fd32c074410 0x7fd32c0768b0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:08.416 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.415+0000 7fd33dffb640 10 client.17849.objecter ms_handle_connect 0x7fd32c074410 2023-12-15T17:24:08.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.439+0000 7fd33dffb640 1 -- 172.21.15.69:0/60907252 <== osd.7 v2:172.21.15.192:6826/4283271495 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd32c076f80 con 0x7fd32c074410 2023-12-15T17:24:08.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.439+0000 7fd33dffb640 10 client.17849.objecter ms_dispatch 0x7fd348000b90 command_reply(tid 1: 0 ) v1 2023-12-15T17:24:08.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.439+0000 7fd33dffb640 10 client.17849.objecter _finish_command 1 = system:0 2023-12-15T17:24:08.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.439+0000 7fd33dffb640 20 client.17849.objecter put_session s=0x7fd32c073d40 osd=7 4 2023-12-15T17:24:08.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.439+0000 7fd33dffb640 15 client.17849.objecter _session_command_op_remove 7 1 2023-12-15T17:24:08.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.451+0000 7fd34f832640 10 client.17849.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-15T17:24:08.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.451+0000 7fd34f832640 15 client.17849.objecter _session_command_op_assign -1 2 2023-12-15T17:24:08.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.451+0000 7fd34f832640 20 client.17849.objecter _get_session s=0x7fd32c073d40 osd=7 3 2023-12-15T17:24:08.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.451+0000 7fd34f832640 20 client.17849.objecter put_session s=0x7fd32c073d40 osd=7 3 2023-12-15T17:24:08.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.451+0000 7fd34f832640 20 client.17849.objecter _get_session s=0x7fd32c073d40 osd=7 3 2023-12-15T17:24:08.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.451+0000 7fd34f832640 15 client.17849.objecter _session_command_op_remove -1 2 2023-12-15T17:24:08.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.451+0000 7fd34f832640 20 client.17849.objecter get_session s=0x7fd32c073d40 osd=7 3 2023-12-15T17:24:08.452 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.451+0000 7fd34f832640 15 client.17849.objecter _session_command_op_assign 7 2 2023-12-15T17:24:08.453 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.451+0000 7fd34f832640 20 client.17849.objecter put_session s=0x7fd32c073d40 osd=7 4 2023-12-15T17:24:08.453 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.451+0000 7fd34f832640 10 client.17849.objecter _send_command 2 2023-12-15T17:24:08.453 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.451+0000 7fd34f832640 1 -- 172.21.15.69:0/60907252 --> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd348140cf0 con 0x7fd32c074410 2023-12-15T17:24:08.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd33dffb640 1 -- 172.21.15.69:0/60907252 <== osd.7 v2:172.21.15.192:6826/4283271495 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd348140cf0 con 0x7fd32c074410 2023-12-15T17:24:08.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd33dffb640 10 client.17849.objecter ms_dispatch 0x7fd348000b90 command_reply(tid 2: 0 ) v1 2023-12-15T17:24:08.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd33dffb640 10 client.17849.objecter _finish_command 2 = system:0 2023-12-15T17:24:08.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd33dffb640 20 client.17849.objecter put_session s=0x7fd32c073d40 osd=7 4 2023-12-15T17:24:08.465 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd33dffb640 15 client.17849.objecter _session_command_op_remove 7 2 2023-12-15T17:24:08.466 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-12-15T17:24:08.466 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-15T17:24:08.466 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-12-15T17:24:08.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 10 librados: watch_flush enter 2023-12-15T17:24:08.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 10 librados: watch_flush exit 2023-12-15T17:24:08.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 10 client.17849.objecter close_session for osd.7 2023-12-15T17:24:08.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 1 -- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] conn(0x7fd32c074410 msgr2=0x7fd32c0768b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:6826/4283271495,v1:172.21.15.192:6827/4283271495] conn(0x7fd32c074410 0x7fd32c0768b0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 20 client.17849.objecter put_session s=0x7fd32c073d40 osd=7 1 2023-12-15T17:24:08.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 20 client.17849.objecter shutdown clearing up homeless session... 2023-12-15T17:24:08.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 10 client.17849.objecter successfully canceled tick 2023-12-15T17:24:08.467 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 1 -- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fd32c06cb80 msgr2=0x7fd32c06f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7fd32c06cb80 0x7fd32c06f040 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd338000f60 tx=0x7fd3380079b0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 10 monclient: shutdown 2023-12-15T17:24:08.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:08.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 1 -- 172.21.15.69:0/60907252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34813f7c0 msgr2=0x7fd3481347e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 1 --2- 172.21.15.69:0/60907252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34813f7c0 0x7fd3481347e0 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7fd34400e8d0 tx=0x7fd34400ed90 comp rx=0 tx=0).stop 2023-12-15T17:24:08.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34ddcf640 1 -- 172.21.15.69:0/60907252 reap_dead start 2023-12-15T17:24:08.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 1 -- 172.21.15.69:0/60907252 shutdown_connections 2023-12-15T17:24:08.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 1 -- 172.21.15.69:0/60907252 >> 172.21.15.69:0/60907252 conn(0x7fd3480619a0 msgr2=0x7fd34813df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:08.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 1 -- 172.21.15.69:0/60907252 shutdown_connections 2023-12-15T17:24:08.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 1 -- 172.21.15.69:0/60907252 wait complete. 2023-12-15T17:24:08.468 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.465+0000 7fd34f832640 1 librados: shutdown 2023-12-15T17:24:08.476 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-15T17:24:08.575 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.574+0000 7f7b5d1cc640 10 monclient: get_monmap_and_config 2023-12-15T17:24:08.575 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.574+0000 7f7b5d1cc640 10 monclient: build_initial_monmap 2023-12-15T17:24:08.575 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.574+0000 7f7b5d1cc640 10 monclient: monmap: 2023-12-15T17:24:08.575 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:08.575 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:08.575 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:08.575209+0000 2023-12-15T17:24:08.575 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:08.575209+0000 2023-12-15T17:24:08.575 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:08.575 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.575 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:08.576 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:08.576 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:08.576 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:08.576 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:08.576 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.575+0000 7f7b5d1cc640 1 Processor -- start 2023-12-15T17:24:08.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.575+0000 7f7b5d1cc640 1 -- start start 2023-12-15T17:24:08.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.575+0000 7f7b5d1cc640 10 monclient: init 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.575+0000 7f7b5d1cc640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.575+0000 7f7b5d1cc640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.575+0000 7f7b5d1cc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b580a8a30 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.575+0000 7f7b5d1cc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7b58142bf0 0x7f7b5813ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.575+0000 7f7b5d1cc640 10 monclient(hunting): picked mon.noname-c con 0x7f7b58142bf0 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5d1cc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b580a8e10 0x7f7b5813f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5d1cc640 10 monclient(hunting): picked mon.noname-a con 0x7f7b580a8e10 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5d1cc640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b580aac10 con 0x7f7b580a8a30 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5d1cc640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5d1cc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b580aaa10 con 0x7f7b580a8e10 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5d1cc640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5d1cc640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b580aaef0 con 0x7f7b58142bf0 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5d1cc640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:08.578 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5d1cc640 10 monclient(hunting): authenticate will time out at 2752.891416s 2023-12-15T17:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5759e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b580a8e10 0x7f7b5813f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5759e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b580a8e10 0x7f7b5813f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53384/0 (socket says 172.21.15.69:53384) 2023-12-15T17:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5759e640 1 -- 172.21.15.69:0/2313674601 learned_addr learned my addr 172.21.15.69:0/2313674601 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5759e640 10 monclient(hunting): get_auth_request con 0x7f7b580a8e10 auth_method 0 2023-12-15T17:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5759e640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.576+0000 7f7b5759e640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.577+0000 7f7b5759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.577+0000 7f7b5759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.577+0000 7f7b5759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:08.579 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.577+0000 7f7b5659c640 1 --2- 172.21.15.69:0/2313674601 >> v2:172.21.15.192:3300/0 conn(0x7f7b58142bf0 0x7f7b5813ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:08.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5659c640 10 monclient(hunting): get_auth_request con 0x7f7b58142bf0 auth_method 0 2023-12-15T17:24:08.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:08.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5659c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:08.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:08.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:08.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:08.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:08.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5759e640 10 monclient(hunting): handle_auth_done global_id 16131 payload 274 2023-12-15T17:24:08.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5759e640 1 -- 172.21.15.69:0/2313674601 >> v2:172.21.15.192:3300/0 conn(0x7f7b58142bf0 msgr2=0x7f7b5813ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:08.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5759e640 1 --2- 172.21.15.69:0/2313674601 >> v2:172.21.15.192:3300/0 conn(0x7f7b58142bf0 0x7f7b5813ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5759e640 1 -- 172.21.15.69:0/2313674601 >> v1:172.21.15.192:6789/0 conn(0x7f7b580a8a30 legacy=0x7f7b581424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:08.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5759e640 10 monclient: _finish_hunting 0 2023-12-15T17:24:08.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5759e640 1 monclient: found mon.noname-a 2023-12-15T17:24:08.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:08.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:24:08.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.578+0000 7f7b5759e640 1 -- 172.21.15.69:0/2313674601 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b580a97f0 con 0x7f7b580a8e10 2023-12-15T17:24:08.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.580+0000 7f7b5759e640 1 --2- 172.21.15.69:0/2313674601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b580a8e10 0x7f7b5813f230 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7f7b580b87d0 tx=0x7f7b5002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac4e8915ff10f0a server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:08.584 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 1 -- 172.21.15.69:0/2313674601 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5002cb20 con 0x7f7b580a8e10 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 10 monclient: dump: 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 10 monclient: _finish_auth 0 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 10 monclient: _check_auth_tickets 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 1 -- 172.21.15.69:0/2313674601 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5002cc80 con 0x7f7b580a8e10 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 1 -- 172.21.15.69:0/2313674601 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b500343f0 con 0x7f7b580a8e10 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b55d9b640 10 monclient: dump: 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.586 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:08.587 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:08.587 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:08.587 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b5d1cc640 5 monclient: authenticate success, global_id 16131 2023-12-15T17:24:08.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b5d1cc640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:08.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.584+0000 7f7b5d1cc640 4 set_mon_vals no callback set 2023-12-15T17:24:08.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.585+0000 7f7b5d1cc640 10 monclient: shutdown 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.585+0000 7f7b5d1cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.585+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/2313674601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b580a8e10 msgr2=0x7f7b5813f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.585+0000 7f7b5d1cc640 1 --2- 172.21.15.69:0/2313674601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b580a8e10 0x7f7b5813f230 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7f7b580b87d0 tx=0x7f7b5002bf80 comp rx=0 tx=0).stop 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.585+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/2313674601 shutdown_connections 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.585+0000 7f7b5d1cc640 1 --2- 172.21.15.69:0/2313674601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b580a8e10 0x7f7b5813f230 unknown :-1 s=CLOSED pgs=3166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.585+0000 7f7b5d1cc640 1 --2- 172.21.15.69:0/2313674601 >> v2:172.21.15.192:3300/0 conn(0x7f7b58142bf0 0x7f7b5813ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.585+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/2313674601 >> 172.21.15.69:0/2313674601 conn(0x7f7b58061a30 msgr2=0x7f7b580b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.585+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/2313674601 shutdown_connections 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.585+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/2313674601 wait complete. 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.586+0000 7f7b5d1cc640 10 monclient: build_initial_monmap 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.586+0000 7f7b5d1cc640 10 monclient: monmap: 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:08.590 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.586+0000 7f7b5d1cc640 1 librados: starting msgr at 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.586+0000 7f7b5d1cc640 1 librados: starting objecter 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.586+0000 7f7b5d1cc640 1 Processor -- start 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.586+0000 7f7b5d1cc640 1 -- start start 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 1 librados: setting wanted keys 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 1 librados: calling monclient init 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 10 monclient: init 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:08.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b580a8a30 0x7f7b580aba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 10 monclient(hunting): picked mon.noname-a con 0x7f7b580a8a30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b580a8e10 0x7f7b580b0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 10 monclient(hunting): picked mon.noname-c con 0x7f7b580a8e10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58142bf0 0x7f7b580b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 10 monclient(hunting): picked mon.noname-b con 0x7f7b58142bf0 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58145dd0 con 0x7f7b580a8a30 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b580abfb0 con 0x7f7b58142bf0 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b580ac170 con 0x7f7b580a8e10 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b5d1cc640 10 monclient(hunting): authenticate will time out at 2752.902886s 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.587+0000 7f7b56d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b580a8a30 0x7f7b580aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b5759e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58142bf0 0x7f7b580b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b5759e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58142bf0 0x7f7b580b0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.69:47100/0 (socket says 172.21.15.69:47100) 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b56d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b580a8a30 0x7f7b580aba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53398/0 (socket says 172.21.15.69:53398) 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b5759e640 1 -- 172.21.15.69:0/67455485 learned_addr learned my addr 172.21.15.69:0/67455485 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b5759e640 10 monclient(hunting): get_auth_request con 0x7f7b58142bf0 auth_method 0 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b5759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b5759e640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b5759e640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b56d9d640 10 monclient(hunting): get_auth_request con 0x7f7b580a8a30 auth_method 0 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b56d9d640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b56d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b5759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b5759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:08.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b5759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:08.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:08.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:08.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.588+0000 7f7b56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:08.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.589+0000 7f7b5659c640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b580a8e10 0x7f7b580b0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:08.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.589+0000 7f7b5659c640 10 monclient(hunting): get_auth_request con 0x7f7b580a8e10 auth_method 0 2023-12-15T17:24:08.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.589+0000 7f7b5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:08.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.589+0000 7f7b5659c640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:08.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.589+0000 7f7b5659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.590+0000 7f7b5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.590+0000 7f7b5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.590+0000 7f7b5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.591+0000 7f7b5759e640 10 monclient(hunting): handle_auth_done global_id 21511 payload 824 2023-12-15T17:24:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.591+0000 7f7b5759e640 1 -- 172.21.15.69:0/67455485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b580a8e10 msgr2=0x7f7b580b0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.591+0000 7f7b5759e640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b580a8e10 0x7f7b580b0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.591+0000 7f7b5759e640 1 -- 172.21.15.69:0/67455485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b580a8a30 msgr2=0x7f7b580aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.591+0000 7f7b5759e640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b580a8a30 0x7f7b580aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.591+0000 7f7b5759e640 10 monclient: _finish_hunting 0 2023-12-15T17:24:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.591+0000 7f7b5759e640 1 monclient: found mon.noname-b 2023-12-15T17:24:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.591+0000 7f7b5759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:08.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.591+0000 7f7b5759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:08.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.591+0000 7f7b5759e640 1 -- 172.21.15.69:0/67455485 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b50009b30 con 0x7f7b58142bf0 2023-12-15T17:24:08.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.593+0000 7f7b5759e640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58142bf0 0x7f7b580b0b60 secure :-1 s=READY pgs=5129 cs=0 l=1 rev1=1 crypto rx=0x7f7b50036040 tx=0x7f7b50004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:08.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.593+0000 7f7b3f7fe640 10 client.?.objecter ms_handle_connect 0x7f7b58142bf0 2023-12-15T17:24:08.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.593+0000 7f7b3f7fe640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:08.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 1 -- 172.21.15.69:0/67455485 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5002c920 con 0x7f7b58142bf0 2023-12-15T17:24:08.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:08.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:08.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 10 monclient: dump: 2023-12-15T17:24:08.598 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:08.598 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:08.598 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 10 monclient: _finish_auth 0 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 10 monclient: _check_auth_tickets 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 1 -- 172.21.15.69:0/67455485 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5002caa0 con 0x7f7b58142bf0 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b5d1cc640 5 monclient: authenticate success, global_id 21511 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b5d1cc640 10 monclient: _renew_subs 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b5d1cc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:08.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/67455485 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b580b10a0 con 0x7f7b58142bf0 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 1 -- 172.21.15.69:0/67455485 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5003e690 con 0x7f7b58142bf0 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b3f7fe640 10 monclient: dump: 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:08.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b5559a640 4 set_mon_vals no callback set 2023-12-15T17:24:08.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b5d1cc640 10 client.21511.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:08.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b5d1cc640 10 monclient: _renew_subs 2023-12-15T17:24:08.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b5d1cc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:08.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/67455485 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b580ac820 con 0x7f7b58142bf0 2023-12-15T17:24:08.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.597+0000 7f7b5d1cc640 1 librados: init done 2023-12-15T17:24:08.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.598+0000 7f7b5d1cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:08.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.598+0000 7f7b5d1cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:08.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.598+0000 7f7b5d1cc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:08.601 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.598+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/67455485 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b58067ac0 con 0x7f7b58142bf0 2023-12-15T17:24:08.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.862+0000 7f7b3f7fe640 1 -- 172.21.15.69:0/67455485 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7b5003e7f0 con 0x7f7b58142bf0 2023-12-15T17:24:08.862 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.862+0000 7f7b3f7fe640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7b4406cb50 0x7f7b4406f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:08.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.862+0000 7f7b56d9d640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7b4406cb50 0x7f7b4406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:08.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.862+0000 7f7b56d9d640 10 monclient: get_auth_request con 0x7f7b4406cb50 auth_method 0 2023-12-15T17:24:08.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.863+0000 7f7b3f7fe640 10 client.21511.objecter ms_handle_connect 0x7f7b4406cb50 2023-12-15T17:24:08.863 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.863+0000 7f7b56d9d640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7b4406cb50 0x7f7b4406f010 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7b48004f60 tx=0x7f7b48007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:08.865 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.865+0000 7f7b3f7fe640 1 -- 172.21.15.69:0/67455485 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7f7b5007b2c0 con 0x7f7b58142bf0 2023-12-15T17:24:08.865 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.865+0000 7f7b3f7fe640 10 client.21511.objecter ms_dispatch 0x7f7b58000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:08.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.865+0000 7f7b3f7fe640 3 client.21511.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:08.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.865+0000 7f7b3f7fe640 3 client.21511.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:08.866 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.865+0000 7f7b3f7fe640 20 client.21511.objecter dump_active .. 0 homeless 2023-12-15T17:24:08.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.875+0000 7f7b3f7fe640 1 -- 172.21.15.69:0/67455485 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7b50077600 con 0x7f7b58142bf0 2023-12-15T17:24:08.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.875+0000 7f7b3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:08.875 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:08.875+0000 7f7b3f7fe640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:09.049 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.048+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/67455485 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b580acf70 con 0x7f7b4406cb50 2023-12-15T17:24:09.050 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.049+0000 7f7b3f7fe640 1 -- 172.21.15.69:0/67455485 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7b580acf70 con 0x7f7b4406cb50 2023-12-15T17:24:09.050 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 7 to deep-scrub 2023-12-15T17:24:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 10 librados: watch_flush enter 2023-12-15T17:24:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 10 librados: watch_flush exit 2023-12-15T17:24:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 20 client.21511.objecter shutdown clearing up homeless session... 2023-12-15T17:24:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 10 client.21511.objecter successfully canceled tick 2023-12-15T17:24:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/67455485 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7b4406cb50 msgr2=0x7f7b4406f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7b4406cb50 0x7f7b4406f010 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7b48004f60 tx=0x7f7b48007e30 comp rx=0 tx=0).stop 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 10 monclient: shutdown 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/67455485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58142bf0 msgr2=0x7f7b580b0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58142bf0 0x7f7b580b0b60 secure :-1 s=READY pgs=5129 cs=0 l=1 rev1=1 crypto rx=0x7f7b50036040 tx=0x7f7b50004a40 comp rx=0 tx=0).stop 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/67455485 shutdown_connections 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f7b4406cb50 0x7f7b4406f010 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58142bf0 0x7f7b580b0b60 unknown :-1 s=CLOSED pgs=5129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b580a8e10 0x7f7b580b0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 --2- 172.21.15.69:0/67455485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b580a8a30 0x7f7b580aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/67455485 >> 172.21.15.69:0/67455485 conn(0x7f7b58061a30 msgr2=0x7f7b5813cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/67455485 shutdown_connections 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 -- 172.21.15.69:0/67455485 wait complete. 2023-12-15T17:24:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.051+0000 7f7b5d1cc640 1 librados: shutdown 2023-12-15T17:24:09.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T17:24:09.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.161+0000 7ff9d9ca2640 10 monclient: get_monmap_and_config 2023-12-15T17:24:09.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.161+0000 7ff9d9ca2640 10 monclient: build_initial_monmap 2023-12-15T17:24:09.161 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.161+0000 7ff9d9ca2640 10 monclient: monmap: 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:09.161557+0000 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:09.161557+0000 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:09.162 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:09.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.161+0000 7ff9d9ca2640 1 Processor -- start 2023-12-15T17:24:09.164 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.161+0000 7ff9d9ca2640 1 -- start start 2023-12-15T17:24:09.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.161+0000 7ff9d9ca2640 10 monclient: init 2023-12-15T17:24:09.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:09.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-12-15T17:24:09.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9d40a8910 0x7ff9d413ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:09.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 10 monclient(hunting): picked mon.noname-a con 0x7ff9d40a8910 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:09.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff9d40a9b70 0x7ff9d413f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:09.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 10 monclient(hunting): picked mon.noname-c con 0x7ff9d40a9b70 addr v2:172.21.15.192:3300/0 2023-12-15T17:24:09.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d4141f90 0x7ff9d4142440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:09.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 10 monclient(hunting): picked mon.noname-b con 0x7ff9d4141f90 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:09.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:09.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d40a8cf0 con 0x7ff9d40a8910 2023-12-15T17:24:09.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:09.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d413f6e0 con 0x7ff9d40a9b70 2023-12-15T17:24:09.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:09.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40aabd0 con 0x7ff9d4141f90 2023-12-15T17:24:09.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:09.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.162+0000 7ff9d9ca2640 10 monclient(hunting): authenticate will time out at 2753.477886s 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d37fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9d40a8910 0x7ff9d413ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d4141f90 0x7ff9d4142440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9caffd640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff9d40a9b70 0x7ff9d413f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d37fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9d40a8910 0x7ff9d413ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53400/0 (socket says 172.21.15.69:53400) 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d37fe640 1 -- 172.21.15.69:0/1244764372 learned_addr learned my addr 172.21.15.69:0/1244764372 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d3fff640 10 monclient(hunting): get_auth_request con 0x7ff9d4141f90 auth_method 0 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d3fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d37fe640 10 monclient(hunting): get_auth_request con 0x7ff9d40a8910 auth_method 0 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d37fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9caffd640 10 monclient(hunting): get_auth_request con 0x7ff9d40a9b70 auth_method 0 2023-12-15T17:24:09.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:09.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9caffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:09.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9caffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:09.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:09.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:09.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.163+0000 7ff9d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:09.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.164+0000 7ff9d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:09.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.164+0000 7ff9d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:09.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.164+0000 7ff9d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:09.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.164+0000 7ff9caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:09.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.164+0000 7ff9caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:09.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.164+0000 7ff9caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:09.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9d37fe640 10 monclient(hunting): handle_auth_done global_id 16137 payload 274 2023-12-15T17:24:09.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9d37fe640 1 -- 172.21.15.69:0/1244764372 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d4141f90 msgr2=0x7ff9d4142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:09.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9d37fe640 1 --2- 172.21.15.69:0/1244764372 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d4141f90 0x7ff9d4142440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9d37fe640 1 -- 172.21.15.69:0/1244764372 >> v2:172.21.15.192:3300/0 conn(0x7ff9d40a9b70 msgr2=0x7ff9d413f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:09.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9d37fe640 1 --2- 172.21.15.69:0/1244764372 >> v2:172.21.15.192:3300/0 conn(0x7ff9d40a9b70 0x7ff9d413f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9d37fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:09.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9d37fe640 1 monclient: found mon.noname-a 2023-12-15T17:24:09.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:09.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9d37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:24:09.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9d37fe640 1 -- 172.21.15.69:0/1244764372 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d40aaeb0 con 0x7ff9d40a8910 2023-12-15T17:24:09.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9caffd640 1 --2- 172.21.15.69:0/1244764372 >> v2:172.21.15.192:3300/0 conn(0x7ff9d40a9b70 0x7ff9d413f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-15T17:24:09.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.165+0000 7ff9d3fff640 1 --2- 172.21.15.69:0/1244764372 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d4141f90 0x7ff9d4142440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-15T17:24:09.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.167+0000 7ff9d37fe640 1 --2- 172.21.15.69:0/1244764372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d40a8910 0x7ff9d413ec60 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4074590 tx=0x7ff9c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6678c03be947f6b3 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:09.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 1 -- 172.21.15.69:0/1244764372 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4031870 con 0x7ff9d40a8910 2023-12-15T17:24:09.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:09.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:24:09.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 10 monclient: dump: 2023-12-15T17:24:09.171 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:09.171 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:09.171 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:09.171 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:09.171 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 10 monclient: _finish_auth 0 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 1 -- 172.21.15.69:0/1244764372 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c40319d0 con 0x7ff9d40a8910 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 1 -- 172.21.15.69:0/1244764372 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4031d60 con 0x7ff9d40a8910 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d9ca2640 5 monclient: authenticate success, global_id 16137 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d9ca2640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d9ca2640 4 set_mon_vals no callback set 2023-12-15T17:24:09.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.170+0000 7ff9d2ffd640 10 monclient: dump: 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:09.173 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.171+0000 7ff9d9ca2640 10 monclient: shutdown 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.171+0000 7ff9d9ca2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.171+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1244764372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d40a8910 msgr2=0x7ff9d413ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.171+0000 7ff9d9ca2640 1 --2- 172.21.15.69:0/1244764372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d40a8910 0x7ff9d413ec60 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4074590 tx=0x7ff9c402b910 comp rx=0 tx=0).stop 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.171+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1244764372 shutdown_connections 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.171+0000 7ff9d9ca2640 1 --2- 172.21.15.69:0/1244764372 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d4141f90 0x7ff9d4142440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.171+0000 7ff9d9ca2640 1 --2- 172.21.15.69:0/1244764372 >> v2:172.21.15.192:3300/0 conn(0x7ff9d40a9b70 0x7ff9d413f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.171+0000 7ff9d9ca2640 1 --2- 172.21.15.69:0/1244764372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d40a8910 0x7ff9d413ec60 unknown :-1 s=CLOSED pgs=3167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.171+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1244764372 >> 172.21.15.69:0/1244764372 conn(0x7ff9d40b7780 msgr2=0x7ff9d40b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.171+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1244764372 shutdown_connections 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.171+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1244764372 wait complete. 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.172+0000 7ff9d9ca2640 10 monclient: build_initial_monmap 2023-12-15T17:24:09.178 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.172+0000 7ff9d9ca2640 10 monclient: monmap: 2023-12-15T17:24:09.179 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:09.179 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:09.179 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:09.179 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:09.179 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:09.179 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:09.179 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:09.179 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:09.179 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:09.179 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:09.179 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.172+0000 7ff9d9ca2640 1 librados: starting msgr at 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.172+0000 7ff9d9ca2640 1 librados: starting objecter 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.172+0000 7ff9d9ca2640 1 Processor -- start 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.172+0000 7ff9d9ca2640 1 -- start start 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.172+0000 7ff9d9ca2640 1 librados: setting wanted keys 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.172+0000 7ff9d9ca2640 1 librados: calling monclient init 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.172+0000 7ff9d9ca2640 10 monclient: init 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9d40a8910 0x7ff9d41c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 10 monclient(hunting): picked mon.noname-b con 0x7ff9d40a8910 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d4141f90 0x7ff9d40abe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 10 monclient(hunting): picked mon.noname-a con 0x7ff9d4141f90 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d413b930 0x7ff9d413bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 10 monclient(hunting): picked mon.noname-c con 0x7ff9d413b930 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:09.180 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d413c5c0 con 0x7ff9d4141f90 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40ab7d0 con 0x7ff9d40a8910 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d413c320 con 0x7ff9d413b930 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9d9ca2640 10 monclient(hunting): authenticate will time out at 2753.488804s 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.173+0000 7ff9caffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d4141f90 0x7ff9d40abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d413b930 0x7ff9d413bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9caffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d4141f90 0x7ff9d40abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53408/0 (socket says 172.21.15.69:53408) 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9caffd640 1 -- 172.21.15.69:0/1778817115 learned_addr learned my addr 172.21.15.69:0/1778817115 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9caffd640 10 monclient(hunting): get_auth_request con 0x7ff9d4141f90 auth_method 0 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9caffd640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:09.181 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9caffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d3fff640 10 monclient(hunting): get_auth_request con 0x7ff9d413b930 auth_method 0 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d3fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d37fe640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9d40a8910 0x7ff9d41c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d37fe640 10 monclient(hunting): get_auth_request con 0x7ff9d40a8910 auth_method 0 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d37fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:09.182 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.174+0000 7ff9d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.175+0000 7ff9d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.175+0000 7ff9d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.175+0000 7ff9d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.177+0000 7ff9d37fe640 10 monclient(hunting): handle_auth_done global_id 21517 payload 824 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.177+0000 7ff9d37fe640 1 -- 172.21.15.69:0/1778817115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d413b930 msgr2=0x7ff9d413bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.177+0000 7ff9d37fe640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d413b930 0x7ff9d413bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.177+0000 7ff9d37fe640 1 -- 172.21.15.69:0/1778817115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d4141f90 msgr2=0x7ff9d40abe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.177+0000 7ff9d37fe640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d4141f90 0x7ff9d40abe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.177+0000 7ff9d37fe640 10 monclient: _finish_hunting 0 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.177+0000 7ff9d37fe640 1 monclient: found mon.noname-b 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.177+0000 7ff9d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.177+0000 7ff9d37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.192:3300/0 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.177+0000 7ff9d37fe640 1 -- 172.21.15.69:0/1778817115 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc009400 con 0x7ff9d40a8910 2023-12-15T17:24:09.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.177+0000 7ff9caffd640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d4141f90 0x7ff9d40abe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-15T17:24:09.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.179+0000 7ff9d37fe640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9d40a8910 0x7ff9d41c9b90 secure :-1 s=READY pgs=5130 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4033000 tx=0x7ff9c40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:09.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.179+0000 7ff9d0ff9640 10 client.?.objecter ms_handle_connect 0x7ff9d40a8910 2023-12-15T17:24:09.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.179+0000 7ff9d0ff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:09.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 1 -- 172.21.15.69:0/1778817115 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c403ac60 con 0x7ff9d40a8910 2023-12-15T17:24:09.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:09.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:09.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 10 monclient: dump: 2023-12-15T17:24:09.185 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:09.185 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:09.185 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 10 monclient: _finish_auth 0 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 10 monclient: _check_auth_tickets 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d9ca2640 5 monclient: authenticate success, global_id 21517 2023-12-15T17:24:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 1 -- 172.21.15.69:0/1778817115 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c403adc0 con 0x7ff9d40a8910 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d9ca2640 10 monclient: _renew_subs 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d9ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1778817115 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9cc009a70 con 0x7ff9d40a8910 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 1 -- 172.21.15.69:0/1778817115 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4044630 con 0x7ff9d40a8910 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d0ff9640 10 monclient: dump: 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:09.187 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d27fc640 4 set_mon_vals no callback set 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d9ca2640 10 client.21517.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d9ca2640 10 monclient: _renew_subs 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d9ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1778817115 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c40094d0 con 0x7ff9d40a8910 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.183+0000 7ff9d9ca2640 1 librados: init done 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.184+0000 7ff9d9ca2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.184+0000 7ff9d9ca2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.184+0000 7ff9d9ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.192:3300/0 2023-12-15T17:24:09.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.184+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1778817115 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d40ab990 con 0x7ff9d40a8910 2023-12-15T17:24:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.201+0000 7ff9d0ff9640 1 -- 172.21.15.69:0/1778817115 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff9c404d450 con 0x7ff9d40a8910 2023-12-15T17:24:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.202+0000 7ff9d0ff9640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ff9b406cb50 0x7ff9b406f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.202+0000 7ff9caffd640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ff9b406cb50 0x7ff9b406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.202+0000 7ff9caffd640 10 monclient: get_auth_request con 0x7ff9b406cb50 auth_method 0 2023-12-15T17:24:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.202+0000 7ff9d0ff9640 10 client.21517.objecter ms_handle_connect 0x7ff9b406cb50 2023-12-15T17:24:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.202+0000 7ff9caffd640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ff9b406cb50 0x7ff9b406f010 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40abcd0 tx=0x7ff9bc009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:09.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.204+0000 7ff9d0ff9640 1 -- 172.21.15.69:0/1778817115 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7ff9c402d040 con 0x7ff9d40a8910 2023-12-15T17:24:09.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.204+0000 7ff9d0ff9640 10 client.21517.objecter ms_dispatch 0x7ff9d4000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:09.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.204+0000 7ff9d0ff9640 3 client.21517.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:09.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.204+0000 7ff9d0ff9640 3 client.21517.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:09.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.204+0000 7ff9d0ff9640 20 client.21517.objecter dump_active .. 0 homeless 2023-12-15T17:24:09.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.438+0000 7ff9d0ff9640 1 -- 172.21.15.69:0/1778817115 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff9c4076e70 con 0x7ff9d40a8910 2023-12-15T17:24:09.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.438+0000 7ff9d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.438+0000 7ff9d0ff9640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:09.633 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.632+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1778817115 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff9d40b6910 con 0x7ff9b406cb50 2023-12-15T17:24:09.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.637+0000 7ff9d0ff9640 1 -- 172.21.15.69:0/1778817115 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151508 (secure 0 0 0) 0x7ff9d40b6910 con 0x7ff9b406cb50 2023-12-15T17:24:09.639 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:24:09.640 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-12-15T17:24:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.639+0000 7ff9d9ca2640 10 librados: watch_flush enter 2023-12-15T17:24:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 10 librados: watch_flush exit 2023-12-15T17:24:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 20 client.21517.objecter shutdown clearing up homeless session... 2023-12-15T17:24:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 10 client.21517.objecter successfully canceled tick 2023-12-15T17:24:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1778817115 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ff9b406cb50 msgr2=0x7ff9b406f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ff9b406cb50 0x7ff9b406f010 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40abcd0 tx=0x7ff9bc009190 comp rx=0 tx=0).stop 2023-12-15T17:24:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 10 monclient: shutdown 2023-12-15T17:24:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1778817115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9d40a8910 msgr2=0x7ff9d41c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9d40a8910 0x7ff9d41c9b90 secure :-1 s=READY pgs=5130 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4033000 tx=0x7ff9c40046c0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1778817115 shutdown_connections 2023-12-15T17:24:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7ff9b406cb50 0x7ff9b406f010 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d413b930 0x7ff9d413bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d4141f90 0x7ff9d40abe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 --2- 172.21.15.69:0/1778817115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9d40a8910 0x7ff9d41c9b90 unknown :-1 s=CLOSED pgs=5130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1778817115 >> 172.21.15.69:0/1778817115 conn(0x7ff9d40b7780 msgr2=0x7ff9d40b7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1778817115 shutdown_connections 2023-12-15T17:24:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 -- 172.21.15.69:0/1778817115 wait complete. 2023-12-15T17:24:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:09.640+0000 7ff9d9ca2640 1 librados: shutdown 2023-12-15T17:24:09.651 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1583,"stamp":"2023-12-15T17:24:07.783499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":570724,"kb_used_data":45048,"kb_used_omap":12,"kb_used_meta":525555,"kb_avail":754403996,"statfs":{"total":773094113280,"available":772509691904,"internally_reserved":0,"allocated":46129152,"data_stored":24842482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":538168880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":98,"apply_latency_ms":98,"commit_latency_ns":98000000,"apply_latency_ns":98000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.642865"},"pg_stats":[{"pgid":"7.1b","version":"0'0","reported_seq":870,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.321702+0000","last_change":"2023-12-15T17:03:26.413418+0000","last_active":"2023-12-15T17:23:58.321702+0000","last_peered":"2023-12-15T17:23:58.321702+0000","last_clean":"2023-12-15T17:23:58.321702+0000","last_became_active":"2023-12-15T17:03:26.401879+0000","last_became_peered":"2023-12-15T17:03:26.401879+0000","last_unstale":"2023-12-15T17:23:58.321702+0000","last_undegraded":"2023-12-15T17:23:58.321702+0000","last_fullsized":"2023-12-15T17:23:58.321702+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:25:02.124791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":879,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.629232+0000","last_change":"2023-12-15T17:03:43.168165+0000","last_active":"2023-12-15T17:23:58.629232+0000","last_peered":"2023-12-15T17:23:58.629232+0000","last_clean":"2023-12-15T17:23:58.629232+0000","last_became_active":"2023-12-15T17:03:43.159317+0000","last_became_peered":"2023-12-15T17:03:43.159317+0000","last_unstale":"2023-12-15T17:23:58.629232+0000","last_undegraded":"2023-12-15T17:23:58.629232+0000","last_fullsized":"2023-12-15T17:23:58.629232+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":879,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.622193+0000","last_change":"2023-12-15T17:03:27.146666+0000","last_active":"2023-12-15T17:23:58.622193+0000","last_peered":"2023-12-15T17:23:58.622193+0000","last_clean":"2023-12-15T17:23:58.622193+0000","last_became_active":"2023-12-15T17:03:27.137993+0000","last_became_peered":"2023-12-15T17:03:27.137993+0000","last_unstale":"2023-12-15T17:23:58.622193+0000","last_undegraded":"2023-12-15T17:23:58.622193+0000","last_fullsized":"2023-12-15T17:23:58.622193+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":870,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.311497+0000","last_change":"2023-12-15T17:03:32.164840+0000","last_active":"2023-12-15T17:23:58.311497+0000","last_peered":"2023-12-15T17:23:58.311497+0000","last_clean":"2023-12-15T17:23:58.311497+0000","last_became_active":"2023-12-15T17:03:32.155432+0000","last_became_peered":"2023-12-15T17:03:32.155432+0000","last_unstale":"2023-12-15T17:23:58.311497+0000","last_undegraded":"2023-12-15T17:23:58.311497+0000","last_fullsized":"2023-12-15T17:23:58.311497+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:29:00.934002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":856,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:03.958777+0000","last_change":"2023-12-15T17:24:03.959482+0000","last_active":"2023-12-15T17:24:03.958777+0000","last_peered":"2023-12-15T17:24:03.958777+0000","last_clean":"2023-12-15T17:24:03.958777+0000","last_became_active":"2023-12-15T17:03:32.040162+0000","last_became_peered":"2023-12-15T17:03:32.040162+0000","last_unstale":"2023-12-15T17:24:03.958777+0000","last_undegraded":"2023-12-15T17:24:03.958777+0000","last_fullsized":"2023-12-15T17:24:03.958777+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:03.957092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:03.957092+0000","last_clean_scrub_stamp":"2023-12-15T17:24:03.957092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:21:14.132235+0000","scrub_duration":0.071783557999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":855,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.244809+0000","last_change":"2023-12-15T17:03:29.590131+0000","last_active":"2023-12-15T17:23:57.244809+0000","last_peered":"2023-12-15T17:23:57.244809+0000","last_clean":"2023-12-15T17:23:57.244809+0000","last_became_active":"2023-12-15T17:03:29.584729+0000","last_became_peered":"2023-12-15T17:03:29.584729+0000","last_unstale":"2023-12-15T17:23:57.244809+0000","last_undegraded":"2023-12-15T17:23:57.244809+0000","last_fullsized":"2023-12-15T17:23:57.244809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:25:55.347921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":873,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.180768+0000","last_change":"2023-12-15T17:03:27.355790+0000","last_active":"2023-12-15T17:23:58.180768+0000","last_peered":"2023-12-15T17:23:58.180768+0000","last_clean":"2023-12-15T17:23:58.180768+0000","last_became_active":"2023-12-15T17:03:27.349390+0000","last_became_peered":"2023-12-15T17:03:27.349390+0000","last_unstale":"2023-12-15T17:23:58.180768+0000","last_undegraded":"2023-12-15T17:23:58.180768+0000","last_fullsized":"2023-12-15T17:23:58.180768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:50:15.500629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":850,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.251391+0000","last_change":"2023-12-15T17:03:41.190972+0000","last_active":"2023-12-15T17:23:57.251391+0000","last_peered":"2023-12-15T17:23:57.251391+0000","last_clean":"2023-12-15T17:23:57.251391+0000","last_became_active":"2023-12-15T17:03:41.182883+0000","last_became_peered":"2023-12-15T17:03:41.182883+0000","last_unstale":"2023-12-15T17:23:57.251391+0000","last_undegraded":"2023-12-15T17:23:57.251391+0000","last_fullsized":"2023-12-15T17:23:57.251391+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T00:04:20.611993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":853,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.231954+0000","last_change":"2023-12-15T17:03:27.568351+0000","last_active":"2023-12-15T17:23:57.231954+0000","last_peered":"2023-12-15T17:23:57.231954+0000","last_clean":"2023-12-15T17:23:57.231954+0000","last_became_active":"2023-12-15T17:03:27.563090+0000","last_became_peered":"2023-12-15T17:03:27.563090+0000","last_unstale":"2023-12-15T17:23:57.231954+0000","last_undegraded":"2023-12-15T17:23:57.231954+0000","last_fullsized":"2023-12-15T17:23:57.231954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:48:11.443821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":848,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.242448+0000","last_change":"2023-12-15T17:03:26.203421+0000","last_active":"2023-12-15T17:23:57.242448+0000","last_peered":"2023-12-15T17:23:57.242448+0000","last_clean":"2023-12-15T17:23:57.242448+0000","last_became_active":"2023-12-15T17:03:26.168812+0000","last_became_peered":"2023-12-15T17:03:26.168812+0000","last_unstale":"2023-12-15T17:23:57.242448+0000","last_undegraded":"2023-12-15T17:23:57.242448+0000","last_fullsized":"2023-12-15T17:23:57.242448+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:30:33.793828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":850,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.233508+0000","last_change":"2023-12-15T17:03:33.666043+0000","last_active":"2023-12-15T17:23:57.233508+0000","last_peered":"2023-12-15T17:23:57.233508+0000","last_clean":"2023-12-15T17:23:57.233508+0000","last_became_active":"2023-12-15T17:03:33.660278+0000","last_became_peered":"2023-12-15T17:03:33.660278+0000","last_unstale":"2023-12-15T17:23:57.233508+0000","last_undegraded":"2023-12-15T17:23:57.233508+0000","last_fullsized":"2023-12-15T17:23:57.233508+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:21:15.535555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":853,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.228623+0000","last_change":"2023-12-15T17:03:29.499597+0000","last_active":"2023-12-15T17:23:57.228623+0000","last_peered":"2023-12-15T17:23:57.228623+0000","last_clean":"2023-12-15T17:23:57.228623+0000","last_became_active":"2023-12-15T17:03:29.479817+0000","last_became_peered":"2023-12-15T17:03:29.479817+0000","last_unstale":"2023-12-15T17:23:57.228623+0000","last_undegraded":"2023-12-15T17:23:57.228623+0000","last_fullsized":"2023-12-15T17:23:57.228623+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:29:30.677748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":871,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.317995+0000","last_change":"2023-12-15T17:03:27.798402+0000","last_active":"2023-12-15T17:23:58.317995+0000","last_peered":"2023-12-15T17:23:58.317995+0000","last_clean":"2023-12-15T17:23:58.317995+0000","last_became_active":"2023-12-15T17:03:27.764430+0000","last_became_peered":"2023-12-15T17:03:27.764430+0000","last_unstale":"2023-12-15T17:23:58.317995+0000","last_undegraded":"2023-12-15T17:23:58.317995+0000","last_fullsized":"2023-12-15T17:23:58.317995+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:50:51.103963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":885,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.647932+0000","last_change":"2023-12-15T17:02:01.284924+0000","last_active":"2023-12-15T17:23:58.647932+0000","last_peered":"2023-12-15T17:23:58.647932+0000","last_clean":"2023-12-15T17:23:58.647932+0000","last_became_active":"2023-12-15T17:02:01.275938+0000","last_became_peered":"2023-12-15T17:02:01.275938+0000","last_unstale":"2023-12-15T17:23:58.647932+0000","last_undegraded":"2023-12-15T17:23:58.647932+0000","last_fullsized":"2023-12-15T17:23:58.647932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":874,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.177122+0000","last_change":"2023-12-15T17:03:29.741478+0000","last_active":"2023-12-15T17:23:58.177122+0000","last_peered":"2023-12-15T17:23:58.177122+0000","last_clean":"2023-12-15T17:23:58.177122+0000","last_became_active":"2023-12-15T17:03:29.736171+0000","last_became_peered":"2023-12-15T17:03:29.736171+0000","last_unstale":"2023-12-15T17:23:58.177122+0000","last_undegraded":"2023-12-15T17:23:58.177122+0000","last_fullsized":"2023-12-15T17:23:58.177122+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:42:10.131036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":877,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.337344+0000","last_change":"2023-12-15T17:01:59.949551+0000","last_active":"2023-12-15T17:23:58.337344+0000","last_peered":"2023-12-15T17:23:58.337344+0000","last_clean":"2023-12-15T17:23:58.337344+0000","last_became_active":"2023-12-15T17:01:59.918026+0000","last_became_peered":"2023-12-15T17:01:59.918026+0000","last_unstale":"2023-12-15T17:23:58.337344+0000","last_undegraded":"2023-12-15T17:23:58.337344+0000","last_fullsized":"2023-12-15T17:23:58.337344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:10:51.112278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":850,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.232903+0000","last_change":"2023-12-15T17:03:26.256693+0000","last_active":"2023-12-15T17:23:57.232903+0000","last_peered":"2023-12-15T17:23:57.232903+0000","last_clean":"2023-12-15T17:23:57.232903+0000","last_became_active":"2023-12-15T17:03:26.235756+0000","last_became_peered":"2023-12-15T17:03:26.235756+0000","last_unstale":"2023-12-15T17:23:57.232903+0000","last_undegraded":"2023-12-15T17:23:57.232903+0000","last_fullsized":"2023-12-15T17:23:57.232903+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:08:19.767760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":885,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.641962+0000","last_change":"2023-12-15T17:02:01.380902+0000","last_active":"2023-12-15T17:23:58.641962+0000","last_peered":"2023-12-15T17:23:58.641962+0000","last_clean":"2023-12-15T17:23:58.641962+0000","last_became_active":"2023-12-15T17:02:01.375672+0000","last_became_peered":"2023-12-15T17:02:01.375672+0000","last_unstale":"2023-12-15T17:23:58.641962+0000","last_undegraded":"2023-12-15T17:23:58.641962+0000","last_fullsized":"2023-12-15T17:23:58.641962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"32'7","reported_seq":864,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.236020+0000","last_change":"2023-12-15T17:03:26.359500+0000","last_active":"2023-12-15T17:23:57.236020+0000","last_peered":"2023-12-15T17:23:57.236020+0000","last_clean":"2023-12-15T17:23:57.236020+0000","last_became_active":"2023-12-15T17:03:26.343248+0000","last_became_peered":"2023-12-15T17:03:26.343248+0000","last_unstale":"2023-12-15T17:23:57.236020+0000","last_undegraded":"2023-12-15T17:23:57.236020+0000","last_fullsized":"2023-12-15T17:23:57.236020+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:31:17.788444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":881,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.185109+0000","last_change":"2023-12-15T17:01:59.938931+0000","last_active":"2023-12-15T17:23:58.185109+0000","last_peered":"2023-12-15T17:23:58.185109+0000","last_clean":"2023-12-15T17:23:58.185109+0000","last_became_active":"2023-12-15T17:01:59.908680+0000","last_became_peered":"2023-12-15T17:01:59.908680+0000","last_unstale":"2023-12-15T17:23:58.185109+0000","last_undegraded":"2023-12-15T17:23:58.185109+0000","last_fullsized":"2023-12-15T17:23:58.185109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:35:37.366491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":879,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.646360+0000","last_change":"2023-12-15T17:03:27.788302+0000","last_active":"2023-12-15T17:23:58.646360+0000","last_peered":"2023-12-15T17:23:58.646360+0000","last_clean":"2023-12-15T17:23:58.646360+0000","last_became_active":"2023-12-15T17:03:27.780434+0000","last_became_peered":"2023-12-15T17:03:27.780434+0000","last_unstale":"2023-12-15T17:23:58.646360+0000","last_undegraded":"2023-12-15T17:23:58.646360+0000","last_fullsized":"2023-12-15T17:23:58.646360+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":886,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.626127+0000","last_change":"2023-12-15T17:02:01.718405+0000","last_active":"2023-12-15T17:23:58.626127+0000","last_peered":"2023-12-15T17:23:58.626127+0000","last_clean":"2023-12-15T17:23:58.626127+0000","last_became_active":"2023-12-15T17:02:01.713188+0000","last_became_peered":"2023-12-15T17:02:01.713188+0000","last_unstale":"2023-12-15T17:23:58.626127+0000","last_undegraded":"2023-12-15T17:23:58.626127+0000","last_fullsized":"2023-12-15T17:23:58.626127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":871,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.326033+0000","last_change":"2023-12-15T17:03:26.257800+0000","last_active":"2023-12-15T17:23:58.326033+0000","last_peered":"2023-12-15T17:23:58.326033+0000","last_clean":"2023-12-15T17:23:58.326033+0000","last_became_active":"2023-12-15T17:03:26.252650+0000","last_became_peered":"2023-12-15T17:03:26.252650+0000","last_unstale":"2023-12-15T17:23:58.326033+0000","last_undegraded":"2023-12-15T17:23:58.326033+0000","last_fullsized":"2023-12-15T17:23:58.326033+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:20:58.993074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":860,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.222037+0000","last_change":"2023-12-15T17:01:59.559261+0000","last_active":"2023-12-15T17:23:57.222037+0000","last_peered":"2023-12-15T17:23:57.222037+0000","last_clean":"2023-12-15T17:23:57.222037+0000","last_became_active":"2023-12-15T17:01:59.531480+0000","last_became_peered":"2023-12-15T17:01:59.531480+0000","last_unstale":"2023-12-15T17:23:57.222037+0000","last_undegraded":"2023-12-15T17:23:57.222037+0000","last_fullsized":"2023-12-15T17:23:57.222037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:51:07.527384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":879,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.640390+0000","last_change":"2023-12-15T17:03:29.375364+0000","last_active":"2023-12-15T17:23:58.640390+0000","last_peered":"2023-12-15T17:23:58.640390+0000","last_clean":"2023-12-15T17:23:58.640390+0000","last_became_active":"2023-12-15T17:03:29.361304+0000","last_became_peered":"2023-12-15T17:03:29.361304+0000","last_unstale":"2023-12-15T17:23:58.640390+0000","last_undegraded":"2023-12-15T17:23:58.640390+0000","last_fullsized":"2023-12-15T17:23:58.640390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":893,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:07.518690+0000","last_change":"2023-12-15T17:24:07.523169+0000","last_active":"2023-12-15T17:24:07.518690+0000","last_peered":"2023-12-15T17:24:07.518690+0000","last_clean":"2023-12-15T17:24:07.518690+0000","last_became_active":"2023-12-15T17:02:01.208353+0000","last_became_peered":"2023-12-15T17:02:01.208353+0000","last_unstale":"2023-12-15T17:24:07.518690+0000","last_undegraded":"2023-12-15T17:24:07.518690+0000","last_fullsized":"2023-12-15T17:24:07.518690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:07.514766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:07.514766+0000","last_clean_scrub_stamp":"2023-12-15T17:24:07.514766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:25:52.283874+0000","scrub_duration":0.192829009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":874,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.191435+0000","last_change":"2023-12-15T17:03:27.328921+0000","last_active":"2023-12-15T17:23:58.191435+0000","last_peered":"2023-12-15T17:23:58.191435+0000","last_clean":"2023-12-15T17:23:58.191435+0000","last_became_active":"2023-12-15T17:03:27.321338+0000","last_became_peered":"2023-12-15T17:03:27.321338+0000","last_unstale":"2023-12-15T17:23:58.191435+0000","last_undegraded":"2023-12-15T17:23:58.191435+0000","last_fullsized":"2023-12-15T17:23:58.191435+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:51:42.546700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":886,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.634796+0000","last_change":"2023-12-15T17:02:01.625426+0000","last_active":"2023-12-15T17:23:58.634796+0000","last_peered":"2023-12-15T17:23:58.634796+0000","last_clean":"2023-12-15T17:23:58.634796+0000","last_became_active":"2023-12-15T17:02:01.616517+0000","last_became_peered":"2023-12-15T17:02:01.616517+0000","last_unstale":"2023-12-15T17:23:58.634796+0000","last_undegraded":"2023-12-15T17:23:58.634796+0000","last_fullsized":"2023-12-15T17:23:58.634796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:57.969967+0000","last_clean_scrub_stamp":"2023-12-15T17:01:57.969967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":963,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:23:59.067347+0000","last_change":"2023-12-15T17:01:52.907228+0000","last_active":"2023-12-15T17:23:59.067347+0000","last_peered":"2023-12-15T17:23:59.067347+0000","last_clean":"2023-12-15T17:23:59.067347+0000","last_became_active":"2023-12-15T17:01:52.878062+0000","last_became_peered":"2023-12-15T17:01:52.878062+0000","last_unstale":"2023-12-15T17:23:59.067347+0000","last_undegraded":"2023-12-15T17:23:59.067347+0000","last_fullsized":"2023-12-15T17:23:59.067347+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:01:50.252516+0000","last_clean_scrub_stamp":"2023-12-15T17:01:50.252516+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:47:14.717362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":851,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.249866+0000","last_change":"2023-12-15T17:03:31.083758+0000","last_active":"2023-12-15T17:23:57.249866+0000","last_peered":"2023-12-15T17:23:57.249866+0000","last_clean":"2023-12-15T17:23:57.249866+0000","last_became_active":"2023-12-15T17:03:31.075388+0000","last_became_peered":"2023-12-15T17:03:31.075388+0000","last_unstale":"2023-12-15T17:23:57.249866+0000","last_undegraded":"2023-12-15T17:23:57.249866+0000","last_fullsized":"2023-12-15T17:23:57.249866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:32:49.564294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":849,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.247692+0000","last_change":"2023-12-15T17:03:29.701214+0000","last_active":"2023-12-15T17:23:57.247692+0000","last_peered":"2023-12-15T17:23:57.247692+0000","last_clean":"2023-12-15T17:23:57.247692+0000","last_became_active":"2023-12-15T17:03:29.683464+0000","last_became_peered":"2023-12-15T17:03:29.683464+0000","last_unstale":"2023-12-15T17:23:57.247692+0000","last_undegraded":"2023-12-15T17:23:57.247692+0000","last_fullsized":"2023-12-15T17:23:57.247692+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:02:06.165049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":854,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.243247+0000","last_change":"2023-12-15T17:03:26.043197+0000","last_active":"2023-12-15T17:23:57.243247+0000","last_peered":"2023-12-15T17:23:57.243247+0000","last_clean":"2023-12-15T17:23:57.243247+0000","last_became_active":"2023-12-15T17:03:26.036270+0000","last_became_peered":"2023-12-15T17:03:26.036270+0000","last_unstale":"2023-12-15T17:23:57.243247+0000","last_undegraded":"2023-12-15T17:23:57.243247+0000","last_fullsized":"2023-12-15T17:23:57.243247+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:27:28.632284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":852,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.251378+0000","last_change":"2023-12-15T17:03:29.615007+0000","last_active":"2023-12-15T17:23:57.251378+0000","last_peered":"2023-12-15T17:23:57.251378+0000","last_clean":"2023-12-15T17:23:57.251378+0000","last_became_active":"2023-12-15T17:03:29.608805+0000","last_became_peered":"2023-12-15T17:03:29.608805+0000","last_unstale":"2023-12-15T17:23:57.251378+0000","last_undegraded":"2023-12-15T17:23:57.251378+0000","last_fullsized":"2023-12-15T17:23:57.251378+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T20:35:50.230477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":875,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.167794+0000","last_change":"2023-12-15T17:03:27.507868+0000","last_active":"2023-12-15T17:23:58.167794+0000","last_peered":"2023-12-15T17:23:58.167794+0000","last_clean":"2023-12-15T17:23:58.167794+0000","last_became_active":"2023-12-15T17:03:27.499200+0000","last_became_peered":"2023-12-15T17:03:27.499200+0000","last_unstale":"2023-12-15T17:23:58.167794+0000","last_undegraded":"2023-12-15T17:23:58.167794+0000","last_fullsized":"2023-12-15T17:23:58.167794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T20:31:29.233160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":875,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.163182+0000","last_change":"2023-12-15T17:03:28.022217+0000","last_active":"2023-12-15T17:23:58.163182+0000","last_peered":"2023-12-15T17:23:58.163182+0000","last_clean":"2023-12-15T17:23:58.163182+0000","last_became_active":"2023-12-15T17:03:28.012736+0000","last_became_peered":"2023-12-15T17:03:28.012736+0000","last_unstale":"2023-12-15T17:23:58.163182+0000","last_undegraded":"2023-12-15T17:23:58.163182+0000","last_fullsized":"2023-12-15T17:23:58.163182+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T23:37:05.516314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":871,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.316221+0000","last_change":"2023-12-15T17:03:27.802217+0000","last_active":"2023-12-15T17:23:58.316221+0000","last_peered":"2023-12-15T17:23:58.316221+0000","last_clean":"2023-12-15T17:23:58.316221+0000","last_became_active":"2023-12-15T17:03:27.791224+0000","last_became_peered":"2023-12-15T17:03:27.791224+0000","last_unstale":"2023-12-15T17:23:58.316221+0000","last_undegraded":"2023-12-15T17:23:58.316221+0000","last_fullsized":"2023-12-15T17:23:58.316221+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:54:03.758840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":851,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.326424+0000","last_change":"2023-12-15T17:03:34.018076+0000","last_active":"2023-12-15T17:23:58.326424+0000","last_peered":"2023-12-15T17:23:58.326424+0000","last_clean":"2023-12-15T17:23:58.326424+0000","last_became_active":"2023-12-15T17:03:33.897262+0000","last_became_peered":"2023-12-15T17:03:33.897262+0000","last_unstale":"2023-12-15T17:23:58.326424+0000","last_undegraded":"2023-12-15T17:23:58.326424+0000","last_fullsized":"2023-12-15T17:23:58.326424+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T23:33:26.034906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":874,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.183460+0000","last_change":"2023-12-15T17:03:28.235706+0000","last_active":"2023-12-15T17:23:58.183460+0000","last_peered":"2023-12-15T17:23:58.183460+0000","last_clean":"2023-12-15T17:23:58.183460+0000","last_became_active":"2023-12-15T17:03:28.164968+0000","last_became_peered":"2023-12-15T17:03:28.164968+0000","last_unstale":"2023-12-15T17:23:58.183460+0000","last_undegraded":"2023-12-15T17:23:58.183460+0000","last_fullsized":"2023-12-15T17:23:58.183460+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:55:52.108198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":878,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:23:58.637788+0000","last_change":"2023-12-15T17:03:29.256120+0000","last_active":"2023-12-15T17:23:58.637788+0000","last_peered":"2023-12-15T17:23:58.637788+0000","last_clean":"2023-12-15T17:23:58.637788+0000","last_became_active":"2023-12-15T17:03:29.219241+0000","last_became_peered":"2023-12-15T17:03:29.219241+0000","last_unstale":"2023-12-15T17:23:58.637788+0000","last_undegraded":"2023-12-15T17:23:58.637788+0000","last_fullsized":"2023-12-15T17:23:58.637788+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":848,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.254147+0000","last_change":"2023-12-15T17:03:25.431259+0000","last_active":"2023-12-15T17:23:57.254147+0000","last_peered":"2023-12-15T17:23:57.254147+0000","last_clean":"2023-12-15T17:23:57.254147+0000","last_became_active":"2023-12-15T17:03:25.394333+0000","last_became_peered":"2023-12-15T17:03:25.394333+0000","last_unstale":"2023-12-15T17:23:57.254147+0000","last_undegraded":"2023-12-15T17:23:57.254147+0000","last_fullsized":"2023-12-15T17:23:57.254147+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:35:32.135097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":848,"reported_epoch":906,"state":"active+clean","last_fresh":"2023-12-15T17:23:57.255792+0000","last_change":"2023-12-15T17:03:25.933659+0000","last_active":"2023-12-15T17:23:57.255792+0000","last_peered":"2023-12-15T17:23:57.255792+0000","last_clean":"2023-12-15T17:23:57.255792+0000","last_became_active":"2023-12-15T17:03:25.924138+0000","last_became_peered":"2023-12-15T17:03:25.924138+0000","last_unstale":"2023-12-15T17:23:57.255792+0000","last_undegraded":"2023-12-15T17:23:57.255792+0000","last_fullsized":"2023-12-15T17:23:57.255792+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:03:21.572926+0000","last_clean_scrub_stamp":"2023-12-15T17:03:21.572926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:30:47.546645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58924,"kb_used_data":5536,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312916,"statfs":{"total":96636764160,"available":96576425984,"internally_reserved":0,"allocated":5668864,"data_stored":3005854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":10.289999999999999,"5min":13.026999999999999,"15min":16.065000000000001},"min":{"1min":5.0090000000000003,"5min":4.351,"15min":4.3390000000000004},"max":{"1min":20.521000000000001,"5min":151.61099999999999,"15min":199.32400000000001},"last":5.8239999999999998},{"interface":"front","average":{"1min":10.064,"5min":14.347,"15min":17.850999999999999},"min":{"1min":6.2839999999999998,"5min":5.0149999999999997,"15min":4.9370000000000003},"max":{"1min":20.202999999999999,"5min":278.32799999999997,"15min":330.613},"last":7.0030000000000001}]},{"osd":1,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":16.135000000000002,"5min":17.571999999999999,"15min":19.690000000000001},"min":{"1min":7.4400000000000004,"5min":6.7439999999999998,"15min":5.726},"max":{"1min":65.194999999999993,"5min":159.22399999999999,"15min":303.95400000000001},"last":10.319000000000001},{"interface":"front","average":{"1min":15.039999999999999,"5min":19.803000000000001,"15min":20.053999999999998},"min":{"1min":7.5439999999999996,"5min":6.923,"15min":6.4610000000000003},"max":{"1min":55.744,"5min":326.125,"15min":326.125},"last":10.653}]},{"osd":3,"last update":"Fri Dec 15 17:23:02 2023","interfaces":[{"interface":"back","average":{"1min":10.766999999999999,"5min":17.425999999999998,"15min":23.780000000000001},"min":{"1min":6.6740000000000004,"5min":5.944,"15min":5.9050000000000002},"max":{"1min":30.068000000000001,"5min":309.05399999999997,"15min":313.50400000000002},"last":11.798},{"interface":"front","average":{"1min":15.036,"5min":20.998999999999999,"15min":25.573},"min":{"1min":7.476,"5min":7.2480000000000002,"15min":6.1449999999999996},"max":{"1min":28.457999999999998,"5min":372.85199999999998,"15min":372.85199999999998},"last":11.247999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":15.255000000000001,"5min":23.908000000000001,"15min":24.216000000000001},"min":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":6.5359999999999996},"max":{"1min":26.652000000000001,"5min":397.04000000000002,"15min":397.04000000000002},"last":20.902999999999999},{"interface":"front","average":{"1min":16.533999999999999,"5min":24.268999999999998,"15min":25.311},"min":{"1min":8.5980000000000008,"5min":8.5980000000000008,"15min":7.7809999999999997},"max":{"1min":31.385999999999999,"5min":217.01599999999999,"15min":217.01599999999999},"last":12.808}]},{"osd":5,"last update":"Fri Dec 15 17:23:08 2023","interfaces":[{"interface":"back","average":{"1min":17.076000000000001,"5min":22.579000000000001,"15min":34.334000000000003},"min":{"1min":8.2629999999999999,"5min":8.2629999999999999,"15min":7.7839999999999998},"max":{"1min":29.724,"5min":196.75700000000001,"15min":493.322},"last":13.404},{"interface":"front","average":{"1min":13.827,"5min":18.082000000000001,"15min":30.268000000000001},"min":{"1min":7.5880000000000001,"5min":5.9160000000000004,"15min":5.9160000000000004},"max":{"1min":25.835000000000001,"5min":280.45400000000001,"15min":497.67200000000003},"last":16.933}]},{"osd":6,"last update":"Fri Dec 15 17:23:49 2023","interfaces":[{"interface":"back","average":{"1min":16.161000000000001,"5min":22.856000000000002,"15min":25.489999999999998},"min":{"1min":10.462,"5min":9.4039999999999999,"15min":8.5969999999999995},"max":{"1min":25.347999999999999,"5min":289.68400000000003,"15min":289.68400000000003},"last":18.971},{"interface":"front","average":{"1min":14.811999999999999,"5min":21.321999999999999,"15min":22.838999999999999},"min":{"1min":9.6989999999999998,"5min":7.8200000000000003,"15min":6.9409999999999998},"max":{"1min":22.036000000000001,"5min":274.80099999999999,"15min":274.80099999999999},"last":14.090999999999999}]},{"osd":7,"last update":"Fri Dec 15 17:23:02 2023","interfaces":[{"interface":"back","average":{"1min":19.841999999999999,"5min":24.600999999999999,"15min":31.303000000000001},"min":{"1min":9.3930000000000007,"5min":9.3930000000000007,"15min":8.5820000000000007},"max":{"1min":32.054000000000002,"5min":278.98000000000002,"15min":613.21299999999997},"last":18.007000000000001},{"interface":"front","average":{"1min":17.178999999999998,"5min":22.213999999999999,"15min":29.68},"min":{"1min":7.8360000000000003,"5min":7.8360000000000003,"15min":7.8360000000000003},"max":{"1min":29.925999999999998,"5min":275.72300000000001,"15min":613.91099999999994},"last":19.591999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509711,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81524,"kb_used_data":5928,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290316,"statfs":{"total":96636764160,"available":96553283584,"internally_reserved":0,"allocated":6070272,"data_stored":3403694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":12.414999999999999,"5min":16.533000000000001,"15min":22.98},"min":{"1min":5.2759999999999998,"5min":4.4870000000000001,"15min":4.383},"max":{"1min":36.951999999999998,"5min":94.881,"15min":452.79500000000002},"last":11.606999999999999},{"interface":"front","average":{"1min":16.593,"5min":15.000999999999999,"15min":17.280000000000001},"min":{"1min":5.2960000000000003,"5min":5.0419999999999998,"15min":4.3449999999999998},"max":{"1min":90.009,"5min":127.498,"15min":196.59200000000001},"last":10.978999999999999}]},{"osd":2,"last update":"Fri Dec 15 17:23:03 2023","interfaces":[{"interface":"back","average":{"1min":24.509,"5min":19.314,"15min":25.472000000000001},"min":{"1min":6.4480000000000004,"5min":5.7380000000000004,"15min":5.5190000000000001},"max":{"1min":276.47000000000003,"5min":276.47000000000003,"15min":482.72800000000001},"last":7.0919999999999996},{"interface":"front","average":{"1min":25.614000000000001,"5min":22.815000000000001,"15min":31.640999999999998},"min":{"1min":8.2119999999999997,"5min":6.5629999999999997,"15min":6.1699999999999999},"max":{"1min":261.74400000000003,"5min":261.74400000000003,"15min":454.017},"last":12.288}]},{"osd":3,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":19.303000000000001,"5min":20.509,"15min":28.212},"min":{"1min":7.1100000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"max":{"1min":110.723,"5min":147.37100000000001,"15min":488.23700000000002},"last":12.958},{"interface":"front","average":{"1min":16.472000000000001,"5min":21.302,"15min":26.646000000000001},"min":{"1min":10.347,"5min":9.1549999999999994,"15min":6.4119999999999999},"max":{"1min":30.919,"5min":110.056,"15min":333.53199999999998},"last":16.443000000000001}]},{"osd":4,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":13.672000000000001,"5min":18.148,"15min":26.068999999999999},"min":{"1min":7.1959999999999997,"5min":6.2629999999999999,"15min":5.8319999999999999},"max":{"1min":53.085999999999999,"5min":143.24000000000001,"15min":438.75099999999998},"last":8.782},{"interface":"front","average":{"1min":18.068999999999999,"5min":21.373000000000001,"15min":31.120000000000001},"min":{"1min":10.132999999999999,"5min":8.4510000000000005,"15min":8.4339999999999993},"max":{"1min":41.640000000000001,"5min":78.664000000000001,"15min":443.58300000000003},"last":15.801}]},{"osd":5,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":22.023,"5min":20.832000000000001,"15min":36.176000000000002},"min":{"1min":8.4710000000000001,"5min":6.8369999999999997,"15min":6.8369999999999997},"max":{"1min":112.105,"5min":113.04300000000001,"15min":451.70999999999998},"last":13.555999999999999},{"interface":"front","average":{"1min":31.739999999999998,"5min":27.744,"15min":29.515999999999998},"min":{"1min":12.116,"5min":9.8840000000000003,"15min":6.6500000000000004},"max":{"1min":209.96600000000001,"5min":209.96600000000001,"15min":264.46899999999999},"last":17.594999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":21.939,"5min":22.827999999999999,"15min":31.187999999999999},"min":{"1min":12.606,"5min":8.6809999999999992,"15min":7.5629999999999997},"max":{"1min":90.926000000000002,"5min":162.06200000000001,"15min":455.43000000000001},"last":14.193},{"interface":"front","average":{"1min":19.638999999999999,"5min":21.096,"15min":28.334},"min":{"1min":9.1400000000000006,"5min":7.3399999999999999,"15min":6.7249999999999996},"max":{"1min":94.715999999999994,"5min":169.34,"15min":314.589},"last":9.9640000000000004}]},{"osd":7,"last update":"Fri Dec 15 17:23:17 2023","interfaces":[{"interface":"back","average":{"1min":21.369,"5min":26.395,"15min":36.750999999999998},"min":{"1min":10.898999999999999,"5min":10.691000000000001,"15min":10.561999999999999},"max":{"1min":59.499000000000002,"5min":132.423,"15min":453.04300000000001},"last":19.777999999999999},{"interface":"front","average":{"1min":20.079999999999998,"5min":21.163,"15min":28.721},"min":{"1min":10.175000000000001,"5min":10.175000000000001,"15min":8.6110000000000007},"max":{"1min":63.219999999999999,"5min":77.674000000000007,"15min":239.51499999999999},"last":18.562999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509710,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58916,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312924,"statfs":{"total":96636764160,"available":96576434176,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":13.765000000000001,"5min":13.314,"15min":18.777000000000001},"min":{"1min":6.2359999999999998,"5min":4.3639999999999999,"15min":4.1429999999999998},"max":{"1min":80.552999999999997,"5min":80.552999999999997,"15min":375.77800000000002},"last":10.526},{"interface":"front","average":{"1min":17.832999999999998,"5min":17.524999999999999,"15min":23.969000000000001},"min":{"1min":6.0199999999999996,"5min":4.8170000000000002,"15min":4.8170000000000002},"max":{"1min":94.995999999999995,"5min":94.995999999999995,"15min":556.42700000000002},"last":9.9209999999999994}]},{"osd":2,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":20.041,"5min":19.704999999999998,"15min":21.385999999999999},"min":{"1min":9.3889999999999993,"5min":6.2939999999999996,"15min":6.2939999999999996},"max":{"1min":99.216999999999999,"5min":248.864,"15min":396.13400000000001},"last":13.146000000000001},{"interface":"front","average":{"1min":16.388999999999999,"5min":21.504999999999999,"15min":25.138999999999999},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":5.9790000000000001},"max":{"1min":75.944000000000003,"5min":234.161,"15min":354.649},"last":9.6120000000000001}]},{"osd":3,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":19.158000000000001,"5min":21.491,"15min":22.245000000000001},"min":{"1min":7.6159999999999997,"5min":6.1740000000000004,"15min":6.0830000000000002},"max":{"1min":96.188000000000002,"5min":99.453000000000003,"15min":413.00599999999997},"last":12.417999999999999},{"interface":"front","average":{"1min":15.550000000000001,"5min":19.518999999999998,"15min":24.094999999999999},"min":{"1min":6.0640000000000001,"5min":6.04,"15min":6.04},"max":{"1min":101.57299999999999,"5min":106.357,"15min":526.98599999999999},"last":8.0660000000000007}]},{"osd":4,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":21.193999999999999,"5min":24.096,"15min":24.780999999999999},"min":{"1min":7.1829999999999998,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":119.754,"5min":266.279,"15min":389.75700000000001},"last":16.085999999999999},{"interface":"front","average":{"1min":17.59,"5min":22.911999999999999,"15min":25.393000000000001},"min":{"1min":8.4700000000000006,"5min":8.4700000000000006,"15min":7.3520000000000003},"max":{"1min":37.862000000000002,"5min":227.19200000000001,"15min":436.572},"last":11.861000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":17.655000000000001,"5min":22.420000000000002,"15min":26.187999999999999},"min":{"1min":7.8220000000000001,"5min":7.5359999999999996,"15min":7.3540000000000001},"max":{"1min":67.956999999999994,"5min":121.137,"15min":449.81299999999999},"last":7.7770000000000001},{"interface":"front","average":{"1min":20.068000000000001,"5min":23.791,"15min":27.331},"min":{"1min":10.81,"5min":7.7439999999999998,"15min":6.2539999999999996},"max":{"1min":65.381,"5min":109.90000000000001,"15min":429.75299999999999},"last":17.007999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":19.919,"5min":23.132999999999999,"15min":25.440999999999999},"min":{"1min":10.682,"5min":8.6760000000000002,"15min":7.4699999999999998},"max":{"1min":91.619,"5min":93.299000000000007,"15min":469.86799999999999},"last":11.268000000000001},{"interface":"front","average":{"1min":23.565000000000001,"5min":24.164000000000001,"15min":27.797999999999998},"min":{"1min":12.039,"5min":9.2899999999999991,"15min":8.4740000000000002},"max":{"1min":102.664,"5min":102.664,"15min":373.93599999999998},"last":17.898}]},{"osd":7,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":17.504999999999999,"5min":21.597999999999999,"15min":29.437999999999999},"min":{"1min":8.3109999999999999,"5min":7.0449999999999999,"15min":7.0449999999999999},"max":{"1min":77.521000000000001,"5min":151.999,"15min":416.24400000000003},"last":15.44},{"interface":"front","average":{"1min":24.347000000000001,"5min":26.122,"15min":27.091000000000001},"min":{"1min":11.058,"5min":9.8610000000000007,"15min":7.7160000000000002},"max":{"1min":100.611,"5min":163.09700000000001,"15min":410.04300000000001},"last":14.862}]}]},{"osd":3,"up_from":15,"seq":64424509708,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76976,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294864,"statfs":{"total":96636764160,"available":96557940736,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":8.1609999999999996,"5min":13.029,"15min":18.565999999999999},"min":{"1min":5.3899999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"max":{"1min":14.663,"5min":132.791,"15min":252.483},"last":6.9859999999999998},{"interface":"front","average":{"1min":8.7210000000000001,"5min":13.803000000000001,"15min":18.956},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":24.484999999999999,"5min":138.03800000000001,"15min":235.10300000000001},"last":8.0280000000000005}]},{"osd":1,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":10.631,"5min":13.651,"15min":22},"min":{"1min":7.4589999999999996,"5min":4.9390000000000001,"15min":4.9390000000000001},"max":{"1min":19.66,"5min":79.147000000000006,"15min":624.94100000000003},"last":6.7149999999999999},{"interface":"front","average":{"1min":14.163,"5min":17.541,"15min":24.814},"min":{"1min":8.8680000000000003,"5min":7.4770000000000003,"15min":6.9740000000000002},"max":{"1min":22.484000000000002,"5min":86.152000000000001,"15min":635.13199999999995},"last":13.332000000000001}]},{"osd":2,"last update":"Fri Dec 15 17:23:45 2023","interfaces":[{"interface":"back","average":{"1min":14.015000000000001,"5min":17.581,"15min":22.074999999999999},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":60.210000000000001,"5min":122.60899999999999,"15min":286.22699999999998},"last":9.3079999999999998},{"interface":"front","average":{"1min":13.85,"5min":17.390000000000001,"15min":25.114000000000001},"min":{"1min":6.931,"5min":6.931,"15min":6.5380000000000003},"max":{"1min":73.950000000000003,"5min":238.096,"15min":573.05700000000002},"last":9.8949999999999996}]},{"osd":4,"last update":"Fri Dec 15 17:24:04 2023","interfaces":[{"interface":"back","average":{"1min":18.823,"5min":20.178999999999998,"15min":26.625},"min":{"1min":8.5660000000000007,"5min":7.5019999999999998,"15min":7.1159999999999997},"max":{"1min":99.241,"5min":228.708,"15min":628.97299999999996},"last":13.439},{"interface":"front","average":{"1min":22.285,"5min":24.675999999999998,"15min":31.797999999999998},"min":{"1min":12.448,"5min":11.423,"15min":9.9060000000000006},"max":{"1min":97.656000000000006,"5min":230.202,"15min":637.40200000000004},"last":14.025}]},{"osd":5,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":14.532,"5min":20.297999999999998,"15min":26.827999999999999},"min":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.782},"max":{"1min":32.411000000000001,"5min":144.59200000000001,"15min":752.79499999999996},"last":11.813000000000001},{"interface":"front","average":{"1min":16.890999999999998,"5min":21.039000000000001,"15min":25.788},"min":{"1min":9.5,"5min":9.2129999999999992,"15min":6.0119999999999996},"max":{"1min":45.607999999999997,"5min":145.405,"15min":263.79500000000002},"last":10.628}]},{"osd":6,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":16.896000000000001,"5min":21.753,"15min":27.977},"min":{"1min":9.3599999999999994,"5min":8.0150000000000006,"15min":7.5819999999999999},"max":{"1min":40.414999999999999,"5min":134.255,"15min":498.464},"last":15.593999999999999},{"interface":"front","average":{"1min":21.16,"5min":24.161000000000001,"15min":31.126999999999999},"min":{"1min":12.519,"5min":10.362,"15min":8.6560000000000006},"max":{"1min":60.975000000000001,"5min":153.113,"15min":636.16399999999999},"last":8.7479999999999993}]},{"osd":7,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":17.704999999999998,"5min":23.234000000000002,"15min":31.463000000000001},"min":{"1min":11.615,"5min":10.316000000000001,"15min":8.4930000000000003},"max":{"1min":25.109000000000002,"5min":202.89500000000001,"15min":572.08699999999999},"last":12.792},{"interface":"front","average":{"1min":19.895,"5min":24.616,"15min":31.809999999999999},"min":{"1min":13.664999999999999,"5min":13.173999999999999,"15min":10.406000000000001},"max":{"1min":32.581000000000003,"5min":156.91900000000001,"15min":624.04100000000005},"last":15.205}]}]},{"osd":7,"up_from":14,"seq":60129542412,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76968,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294872,"statfs":{"total":96636764160,"available":96557948928,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":11.852,"5min":12.436999999999999,"15min":16.131},"min":{"1min":5.0750000000000002,"5min":4.3810000000000002,"15min":4.3259999999999996},"max":{"1min":37.542000000000002,"5min":91.055999999999997,"15min":161.38499999999999},"last":10.641999999999999},{"interface":"front","average":{"1min":12.800000000000001,"5min":12.664999999999999,"15min":16.463999999999999},"min":{"1min":6.5910000000000002,"5min":4.8339999999999996,"15min":4.8339999999999996},"max":{"1min":64.691999999999993,"5min":93.209999999999994,"15min":142.673},"last":6.7439999999999998}]},{"osd":1,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":20.167000000000002,"5min":18.283000000000001,"15min":22.882000000000001},"min":{"1min":7.2480000000000002,"5min":6.3550000000000004,"15min":5.7629999999999999},"max":{"1min":74.025999999999996,"5min":143.17599999999999,"15min":275.99299999999999},"last":7.4189999999999996},{"interface":"front","average":{"1min":15.464,"5min":14.946,"15min":21.763999999999999},"min":{"1min":5.9749999999999996,"5min":5.7960000000000003,"15min":5.2649999999999997},"max":{"1min":81.099999999999994,"5min":88.022999999999996,"15min":342.22000000000003},"last":8.6010000000000009}]},{"osd":2,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":20.966999999999999,"5min":17.347000000000001,"15min":19.687000000000001},"min":{"1min":8.2509999999999994,"5min":7.383,"15min":6.0670000000000002},"max":{"1min":86.772000000000006,"5min":95.926000000000002,"15min":131.81299999999999},"last":9.4350000000000005},{"interface":"front","average":{"1min":16.437000000000001,"5min":14.654999999999999,"15min":19.41},"min":{"1min":6.9470000000000001,"5min":6.266,"15min":6.266},"max":{"1min":84.876000000000005,"5min":84.876000000000005,"15min":117.864},"last":14.724}]},{"osd":3,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":20.248000000000001,"5min":20.637,"15min":24.939},"min":{"1min":6.5330000000000004,"5min":6.5330000000000004,"15min":6.4370000000000003},"max":{"1min":118.39400000000001,"5min":118.39400000000001,"15min":390.91000000000003},"last":10.023},{"interface":"front","average":{"1min":20.558,"5min":21.027999999999999,"15min":26.327999999999999},"min":{"1min":8.1479999999999997,"5min":7.4260000000000002,"15min":5.7320000000000002},"max":{"1min":138.559,"5min":138.559,"15min":244.13},"last":15.401999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:23:51 2023","interfaces":[{"interface":"back","average":{"1min":24.532,"5min":22.248999999999999,"15min":24.129999999999999},"min":{"1min":7.3529999999999998,"5min":7.3529999999999998,"15min":7.3529999999999998},"max":{"1min":144.52099999999999,"5min":144.52099999999999,"15min":245.65600000000001},"last":11.305},{"interface":"front","average":{"1min":25.827999999999999,"5min":24.245999999999999,"15min":23.055},"min":{"1min":9.4199999999999999,"5min":7.726,"15min":6.9139999999999997},"max":{"1min":143.92099999999999,"5min":189.988,"15min":193.07400000000001},"last":16.030999999999999}]},{"osd":5,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":17.966000000000001,"5min":19.504000000000001,"15min":25.530000000000001},"min":{"1min":7.7699999999999996,"5min":7.7699999999999996,"15min":7.7699999999999996},"max":{"1min":69.656999999999996,"5min":109.459,"15min":198.96000000000001},"last":16.425000000000001},{"interface":"front","average":{"1min":21.565999999999999,"5min":20.701000000000001,"15min":27.978999999999999},"min":{"1min":8.7959999999999994,"5min":7.6189999999999998,"15min":7.6189999999999998},"max":{"1min":145.94,"5min":165.97999999999999,"15min":266.33699999999999},"last":11.917}]},{"osd":6,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":24.082000000000001,"5min":22.300000000000001,"15min":30.507000000000001},"min":{"1min":11.667999999999999,"5min":7.952,"15min":7.8949999999999996},"max":{"1min":112.369,"5min":113.128,"15min":380.62700000000001},"last":17.914000000000001},{"interface":"front","average":{"1min":23.890999999999998,"5min":21.359000000000002,"15min":26.186},"min":{"1min":8.3550000000000004,"5min":8.3550000000000004,"15min":7.2270000000000003},"max":{"1min":108.03,"5min":108.03,"15min":226.91499999999999},"last":14.349}]}]},{"osd":6,"up_from":14,"seq":60129542413,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76968,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294872,"statfs":{"total":96636764160,"available":96557948928,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":10.004,"5min":12.342000000000001,"15min":18.960000000000001},"min":{"1min":4.4269999999999996,"5min":4.133,"15min":4.133},"max":{"1min":25.085999999999999,"5min":128.82499999999999,"15min":223.15199999999999},"last":6.0410000000000004},{"interface":"front","average":{"1min":9.6329999999999991,"5min":14.333,"15min":18.896000000000001},"min":{"1min":6.2009999999999996,"5min":5.3490000000000002,"15min":4.6920000000000002},"max":{"1min":17.561,"5min":120.85299999999999,"15min":190.249},"last":4.8840000000000003}]},{"osd":1,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":12.476000000000001,"5min":18.774000000000001,"15min":22.052},"min":{"1min":7.2279999999999998,"5min":5.1319999999999997,"15min":5.0449999999999999},"max":{"1min":24.771999999999998,"5min":177.15100000000001,"15min":292.72199999999998},"last":11.483000000000001},{"interface":"front","average":{"1min":14.473000000000001,"5min":19.847999999999999,"15min":25.225000000000001},"min":{"1min":7.6120000000000001,"5min":6.4390000000000001,"15min":5.0309999999999997},"max":{"1min":41.270000000000003,"5min":174.02199999999999,"15min":422.36099999999999},"last":9.4700000000000006}]},{"osd":2,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":17.154,"5min":17.030000000000001,"15min":19.913},"min":{"1min":7.7169999999999996,"5min":6.2850000000000001,"15min":5.2699999999999996},"max":{"1min":66.974000000000004,"5min":131.71000000000001,"15min":131.863},"last":12.465999999999999},{"interface":"front","average":{"1min":13.039,"5min":19.218,"15min":23.687000000000001},"min":{"1min":6.9530000000000003,"5min":6.9530000000000003,"15min":6.7439999999999998},"max":{"1min":65.632000000000005,"5min":112.548,"15min":189.417},"last":15.795}]},{"osd":3,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":21.212,"5min":24.562000000000001,"15min":27.657},"min":{"1min":7.7279999999999998,"5min":7.7279999999999998,"15min":7.7279999999999998},"max":{"1min":77.271000000000001,"5min":263.45299999999997,"15min":263.45299999999997},"last":16.436},{"interface":"front","average":{"1min":24.375,"5min":27.704000000000001,"15min":30.245999999999999},"min":{"1min":11.551,"5min":10.195,"15min":6.9100000000000001},"max":{"1min":83.891999999999996,"5min":264.54000000000002,"15min":356.815},"last":17.187000000000001}]},{"osd":4,"last update":"Fri Dec 15 17:23:25 2023","interfaces":[{"interface":"back","average":{"1min":17.318000000000001,"5min":20.620000000000001,"15min":24.326000000000001},"min":{"1min":6.4020000000000001,"5min":6.4020000000000001,"15min":6.4020000000000001},"max":{"1min":118.277,"5min":122.881,"15min":224.96299999999999},"last":13.587},{"interface":"front","average":{"1min":16.988,"5min":21.347000000000001,"15min":26.774000000000001},"min":{"1min":6.9100000000000001,"5min":6.9100000000000001,"15min":6.9100000000000001},"max":{"1min":99.513000000000005,"5min":127.607,"15min":266.88400000000001},"last":12.196999999999999}]},{"osd":5,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":15.765000000000001,"5min":21.884,"15min":28.344000000000001},"min":{"1min":7.9089999999999998,"5min":7.601,"15min":7.0369999999999999},"max":{"1min":40.454999999999998,"5min":164.62899999999999,"15min":349.983},"last":14.804},{"interface":"front","average":{"1min":14.981999999999999,"5min":21.09,"15min":25.739000000000001},"min":{"1min":9.1750000000000007,"5min":7.4530000000000003,"15min":6.8449999999999998},"max":{"1min":26.895,"5min":115.54900000000001,"15min":324.95299999999997},"last":18.181000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:47 2023","interfaces":[{"interface":"back","average":{"1min":15.082000000000001,"5min":22.818999999999999,"15min":28.507000000000001},"min":{"1min":9.4640000000000004,"5min":8.2110000000000003,"15min":6.8520000000000003},"max":{"1min":38.795999999999999,"5min":137.59700000000001,"15min":268.66500000000002},"last":19.98},{"interface":"front","average":{"1min":18.492999999999999,"5min":26.584,"15min":33.052999999999997},"min":{"1min":12.459,"5min":8.8849999999999998,"15min":8.8849999999999998},"max":{"1min":48.68,"5min":135.74199999999999,"15min":318.56900000000002},"last":21.927}]}]},{"osd":5,"up_from":14,"seq":60129542414,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81528,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290312,"statfs":{"total":96636764160,"available":96553279488,"internally_reserved":0,"allocated":6066176,"data_stored":3403689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":10.813000000000001,"5min":18.579999999999998,"15min":19.327999999999999},"min":{"1min":5.0339999999999998,"5min":4.7380000000000004,"15min":4.2859999999999996},"max":{"1min":24.373000000000001,"5min":209.15799999999999,"15min":223.69499999999999},"last":9.1150000000000002},{"interface":"front","average":{"1min":10.352,"5min":17.178999999999998,"15min":19.341999999999999},"min":{"1min":4.2590000000000003,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":17.353000000000002,"5min":183.93899999999999,"15min":226.76300000000001},"last":7.6230000000000002}]},{"osd":1,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":13.563000000000001,"5min":17.940000000000001,"15min":22.196000000000002},"min":{"1min":6.6539999999999999,"5min":5.1260000000000003,"15min":4.7839999999999998},"max":{"1min":63.030999999999999,"5min":137.07499999999999,"15min":340.32999999999998},"last":11.942},{"interface":"front","average":{"1min":18.015999999999998,"5min":20.047999999999998,"15min":24.577000000000002},"min":{"1min":8.7149999999999999,"5min":5.7850000000000001,"15min":5.6550000000000002},"max":{"1min":57.795000000000002,"5min":169.648,"15min":326.375},"last":12.872}]},{"osd":2,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":17.568000000000001,"5min":20.483000000000001,"15min":22.577000000000002},"min":{"1min":6.6630000000000003,"5min":6.6630000000000003,"15min":5.3979999999999997},"max":{"1min":92.230000000000004,"5min":174.09100000000001,"15min":240.00999999999999},"last":10.468999999999999},{"interface":"front","average":{"1min":21.774999999999999,"5min":24.068000000000001,"15min":25.559000000000001},"min":{"1min":6.7690000000000001,"5min":5.569,"15min":5.569},"max":{"1min":128.98099999999999,"5min":228.31899999999999,"15min":371.30099999999999},"last":14.250999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":25.259,"5min":25.724,"15min":28.169},"min":{"1min":9.2319999999999993,"5min":6.5999999999999996,"15min":6.5999999999999996},"max":{"1min":100.711,"5min":177.51300000000001,"15min":321.26100000000002},"last":8.2669999999999995},{"interface":"front","average":{"1min":25.388999999999999,"5min":26.262,"15min":27.506},"min":{"1min":7.3799999999999999,"5min":7.101,"15min":6.8730000000000002},"max":{"1min":101.599,"5min":172.887,"15min":284.12599999999998},"last":18.190000000000001}]},{"osd":4,"last update":"Fri Dec 15 17:23:40 2023","interfaces":[{"interface":"back","average":{"1min":18.481000000000002,"5min":29.667999999999999,"15min":27.547000000000001},"min":{"1min":8.3559999999999999,"5min":8.3559999999999999,"15min":6.8600000000000003},"max":{"1min":61.365000000000002,"5min":229.077,"15min":376.39800000000002},"last":15.064},{"interface":"front","average":{"1min":19.219000000000001,"5min":28.552,"15min":27.09},"min":{"1min":10.989000000000001,"5min":7.8840000000000003,"15min":6.2789999999999999},"max":{"1min":55.509999999999998,"5min":211.03100000000001,"15min":338.56400000000002},"last":11.206}]},{"osd":6,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":25.266999999999999,"15min":27.039000000000001},"min":{"1min":12.606999999999999,"5min":7.8079999999999998,"15min":7.0789999999999997},"max":{"1min":57.170000000000002,"5min":215.21299999999999,"15min":378.327},"last":17.338999999999999},{"interface":"front","average":{"1min":21.053000000000001,"5min":29.216999999999999,"15min":28.443000000000001},"min":{"1min":9.3049999999999997,"5min":8.8900000000000006,"15min":6.6369999999999996},"max":{"1min":62.055999999999997,"5min":228.58799999999999,"15min":407.16500000000002},"last":15.965999999999999}]},{"osd":7,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":19.960999999999999,"5min":24.295000000000002,"15min":28.638999999999999},"min":{"1min":12.951000000000001,"5min":10.864000000000001,"15min":7.7599999999999998},"max":{"1min":39.515999999999998,"5min":151.386,"15min":292.82299999999998},"last":19.015000000000001},{"interface":"front","average":{"1min":20.140000000000001,"5min":27.221,"15min":30.507000000000001},"min":{"1min":11.416,"5min":9.2170000000000005,"15min":7.8650000000000002},"max":{"1min":28.832999999999998,"5min":190.74600000000001,"15min":312.73599999999999},"last":19.876000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542414,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58920,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312920,"statfs":{"total":96636764160,"available":96576430080,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":20.422000000000001,"5min":13.071,"15min":15.058},"min":{"1min":4.7779999999999996,"5min":4.4560000000000004,"15min":4.1289999999999996},"max":{"1min":62.142000000000003,"5min":62.142000000000003,"15min":193.11500000000001},"last":9.6370000000000005},{"interface":"front","average":{"1min":27.135999999999999,"5min":13.522,"15min":16.253},"min":{"1min":5.9699999999999998,"5min":4.4109999999999996,"15min":4.3719999999999999},"max":{"1min":96.224000000000004,"5min":96.224000000000004,"15min":235.09800000000001},"last":7.2370000000000001}]},{"osd":1,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":26.41,"5min":16.433,"15min":19.759},"min":{"1min":8.8640000000000008,"5min":5.2199999999999998,"15min":4.9930000000000003},"max":{"1min":86.936999999999998,"5min":86.936999999999998,"15min":289.11099999999999},"last":12.282},{"interface":"front","average":{"1min":22.891999999999999,"5min":18.584,"15min":21.152999999999999},"min":{"1min":7.3520000000000003,"5min":6.2549999999999999,"15min":6.2249999999999996},"max":{"1min":83.016999999999996,"5min":130.535,"15min":218.32400000000001},"last":8.7029999999999994}]},{"osd":2,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":35.674999999999997,"5min":20.347999999999999,"15min":18.050000000000001},"min":{"1min":8.2569999999999997,"5min":6.8179999999999996,"15min":5.5880000000000001},"max":{"1min":163.31399999999999,"5min":163.31399999999999,"15min":163.31399999999999},"last":10.957000000000001},{"interface":"front","average":{"1min":31.010000000000002,"5min":19.277999999999999,"15min":18.506},"min":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":5.4690000000000003},"max":{"1min":134.78700000000001,"5min":134.78700000000001,"15min":140.416},"last":10.321}]},{"osd":3,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":26.495000000000001,"5min":23.353000000000002,"15min":21.949000000000002},"min":{"1min":11.071999999999999,"5min":8.9169999999999998,"15min":7.2450000000000001},"max":{"1min":89.188000000000002,"5min":230.505,"15min":230.505},"last":12.878},{"interface":"front","average":{"1min":22.827000000000002,"5min":19.329999999999998,"15min":21.117999999999999},"min":{"1min":9.0299999999999994,"5min":6.4210000000000003,"15min":6.4210000000000003},"max":{"1min":84.497,"5min":136.66499999999999,"15min":203.99100000000001},"last":16.268000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":19.713000000000001,"5min":19.503,"15min":21.353999999999999},"min":{"1min":7.7530000000000001,"5min":6.9930000000000003,"15min":6.9930000000000003},"max":{"1min":87.989999999999995,"5min":123.71599999999999,"15min":162.02099999999999},"last":14.503},{"interface":"front","average":{"1min":23.210999999999999,"5min":23.539999999999999,"15min":24.937999999999999},"min":{"1min":10.042999999999999,"5min":10.042999999999999,"15min":7.1130000000000004},"max":{"1min":89.099999999999994,"5min":159.16999999999999,"15min":162.922},"last":18.094000000000001}]},{"osd":6,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":25.030000000000001,"5min":19.335999999999999,"15min":22.289999999999999},"min":{"1min":7.5579999999999998,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":84.040000000000006,"5min":85.606999999999999,"15min":744.33199999999999},"last":11.57},{"interface":"front","average":{"1min":38.479999999999997,"5min":26.033000000000001,"15min":25.606000000000002},"min":{"1min":14.532999999999999,"5min":10.544,"15min":9.1799999999999997},"max":{"1min":157.441,"5min":157.441,"15min":251.19200000000001},"last":17.004999999999999}]},{"osd":7,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":31.425999999999998,"5min":25.463000000000001,"15min":26.760999999999999},"min":{"1min":17.059000000000001,"5min":11.148999999999999,"15min":9.7330000000000005},"max":{"1min":91.006,"5min":144.72,"15min":202.46600000000001},"last":18.681999999999999},{"interface":"front","average":{"1min":21.945,"5min":18.006,"15min":20.908000000000001},"min":{"1min":9.6690000000000005,"5min":7.5860000000000003,"15min":6.5430000000000001},"max":{"1min":78.950000000000003,"5min":85.629999999999995,"15min":199.01400000000001},"last":15.113}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-15T17:24:09.658 INFO:tasks.ceph:pgid 7.1b last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.658 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.658 INFO:tasks.ceph:pgid 7.19 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.659 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.659 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.659 INFO:tasks.ceph:pgid 7.1d last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.659 INFO:tasks.ceph:pgid 7.1c last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.659 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.659 INFO:tasks.ceph:pgid 7.12 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.659 INFO:tasks.ceph:pgid 7.11 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.659 INFO:tasks.ceph:pgid 7.10 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.660 INFO:tasks.ceph:pgid 7.2 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.660 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-15T17:01:57.969967+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=1, tm_sec=57, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.660 INFO:tasks.ceph:pgid 7.3 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.660 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-15T17:01:57.969967+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=1, tm_sec=57, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.660 INFO:tasks.ceph:pgid 7.0 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.660 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-15T17:01:57.969967+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=1, tm_sec=57, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.660 INFO:tasks.ceph:pgid 7.1 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.660 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-15T17:01:57.969967+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=1, tm_sec=57, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.660 INFO:tasks.ceph:pgid 7.7 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.660 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-15T17:01:57.969967+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=1, tm_sec=57, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.661 INFO:tasks.ceph:pgid 7.4 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.661 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-15T17:01:57.969967+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=1, tm_sec=57, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.661 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.661 INFO:tasks.ceph:pgid 7.6 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.661 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-15T17:01:57.969967+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=1, tm_sec=57, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.661 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-15T17:01:50.252516+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=1, tm_sec=50, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.661 INFO:tasks.ceph:pgid 7.c last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.661 INFO:tasks.ceph:pgid 7.d last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.661 INFO:tasks.ceph:pgid 7.e last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.661 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.662 INFO:tasks.ceph:pgid 7.8 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.662 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.662 INFO:tasks.ceph:pgid 7.a last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.662 INFO:tasks.ceph:pgid 7.b last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.662 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.662 INFO:tasks.ceph:pgid 7.15 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.662 INFO:tasks.ceph:pgid 7.16 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.662 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2023-12-15T17:03:21.572926+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=3, tm_sec=21, tm_wday=4, tm_yday=349, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=15, tm_hour=17, tm_min=24, tm_sec=1, tm_wday=4, tm_yday=349, tm_isdst=0) 2023-12-15T17:24:09.663 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-15T17:24:13.722 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:23.667 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:13.725 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:23.671 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:13.741 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:23.687 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:13.769 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:23.708 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:13.773 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:23.713 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:13.777 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:23.723 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:13.778 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:23.724 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:13.786 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:23.726 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:13.797 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:23.742 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:18.744 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:28.690 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:18.746 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:28.692 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:18.758 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:28.704 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:18.791 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:28.731 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:18.792 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:28.738 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:18.792 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:28.732 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:18.794 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:28.739 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:18.802 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:28.742 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:18.806 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:28.752 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:29.663 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T17:24:29.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.762+0000 7f443877c640 10 monclient: get_monmap_and_config 2023-12-15T17:24:29.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.762+0000 7f443877c640 10 monclient: build_initial_monmap 2023-12-15T17:24:29.763 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.762+0000 7f443877c640 10 monclient: monmap: 2023-12-15T17:24:29.763 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:29.763 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:29.763 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:29.763534+0000 2023-12-15T17:24:29.764 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:29.763534+0000 2023-12-15T17:24:29.764 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:29.764 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:29.764 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:29.764 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:29.764 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:29.764 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:29.764 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:29.764 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.762+0000 7f443877c640 1 Processor -- start 2023-12-15T17:24:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.762+0000 7f443877c640 1 -- start start 2023-12-15T17:24:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.762+0000 7f443877c640 10 monclient: init 2023-12-15T17:24:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-12-15T17:24:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44300a8910 addr v1:172.21.15.69:6789/0 2023-12-15T17:24:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a9b70 0x7f443013f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 10 monclient(hunting): picked mon.noname-b con 0x7f44300a9b70 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4430141f90 0x7f44301423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 10 monclient(hunting): picked mon.noname-a con 0x7f4430141f90 addr v2:172.21.15.69:3300/0 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300a8cf0 con 0x7f44300a8910 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44300aabd0 con 0x7f4430141f90 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44300aaeb0 con 0x7f44300a9b70 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.763+0000 7f443877c640 10 monclient(hunting): authenticate will time out at 2774.079888s 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4435d17640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a9b70 0x7f443013f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4436d19640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4430141f90 0x7f44301423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4435d17640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a9b70 0x7f443013f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46200/0 (socket says 172.21.15.69:46200) 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4435d17640 1 -- 172.21.15.69:0/3272580699 learned_addr learned my addr 172.21.15.69:0/3272580699 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4435d17640 10 monclient(hunting): get_auth_request con 0x7f44300a9b70 auth_method 0 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4435d17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4435d17640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4435d17640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4436d19640 10 monclient(hunting): get_auth_request con 0x7f4430141f90 auth_method 0 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4436d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4436d19640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4436d19640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4435d17640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4435d17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.764+0000 7f4435d17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.765+0000 7f4436d19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.765+0000 7f4436d19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:29.768 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.765+0000 7f4436d19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.766+0000 7f4435d17640 10 monclient(hunting): handle_auth_done global_id 17861 payload 274 2023-12-15T17:24:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.766+0000 7f4435d17640 1 -- 172.21.15.69:0/3272580699 >> v2:172.21.15.69:3300/0 conn(0x7f4430141f90 msgr2=0x7f44301423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-15T17:24:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.766+0000 7f4435d17640 1 --2- 172.21.15.69:0/3272580699 >> v2:172.21.15.69:3300/0 conn(0x7f4430141f90 0x7f44301423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.766+0000 7f4435d17640 1 -- 172.21.15.69:0/3272580699 >> v1:172.21.15.69:6789/0 conn(0x7f44300a8910 legacy=0x7f443013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.766+0000 7f4435d17640 10 monclient: _finish_hunting 0 2023-12-15T17:24:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.766+0000 7f4435d17640 1 monclient: found mon.noname-b 2023-12-15T17:24:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.766+0000 7f4435d17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:29.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.766+0000 7f4435d17640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:24:29.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.766+0000 7f4435d17640 1 -- 172.21.15.69:0/3272580699 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44300a9720 con 0x7f44300a9b70 2023-12-15T17:24:29.770 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.768+0000 7f4435d17640 1 --2- 172.21.15.69:0/3272580699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a9b70 0x7f443013f370 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f4420009870 tx=0x7f442002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e33062996e748666 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:29.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 1 -- 172.21.15.69:0/3272580699 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420030840 con 0x7f44300a9b70 2023-12-15T17:24:29.773 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 10 monclient: dump: 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 10 monclient: _finish_auth 0 2023-12-15T17:24:29.774 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 10 monclient: _check_auth_tickets 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 1 -- 172.21.15.69:0/3272580699 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44200309a0 con 0x7f44300a9b70 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f443877c640 5 monclient: authenticate success, global_id 17861 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f443877c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f443877c640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f443877c640 4 set_mon_vals no callback set 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 1 -- 172.21.15.69:0/3272580699 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420030d30 con 0x7f44300a9b70 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.772+0000 7f4435516640 10 monclient: dump: 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:29.775 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:29.776 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:29.776 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:29.776 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:29.776 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:29.776 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:29.776 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:29.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 10 monclient: shutdown 2023-12-15T17:24:29.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:29.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 1 -- 172.21.15.69:0/3272580699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a9b70 msgr2=0x7f443013f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:29.779 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 1 --2- 172.21.15.69:0/3272580699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a9b70 0x7f443013f370 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f4420009870 tx=0x7f442002b980 comp rx=0 tx=0).stop 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 1 -- 172.21.15.69:0/3272580699 shutdown_connections 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 1 --2- 172.21.15.69:0/3272580699 >> v2:172.21.15.69:3300/0 conn(0x7f4430141f90 0x7f44301423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 1 --2- 172.21.15.69:0/3272580699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a9b70 0x7f443013f370 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 1 -- 172.21.15.69:0/3272580699 >> 172.21.15.69:0/3272580699 conn(0x7f44300b7780 msgr2=0x7f44300b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 1 -- 172.21.15.69:0/3272580699 shutdown_connections 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 1 -- 172.21.15.69:0/3272580699 wait complete. 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 10 monclient: build_initial_monmap 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.773+0000 7f443877c640 10 monclient: monmap: 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:29.780 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 librados: starting msgr at 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 librados: starting objecter 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 Processor -- start 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 -- start start 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 librados: setting wanted keys 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 librados: calling monclient init 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 10 monclient: init 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a8910 0x7f4430138600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 10 monclient(hunting): picked mon.noname-c con 0x7f44300a8910 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:29.781 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4430141f90 0x7f443013b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 10 monclient(hunting): picked mon.noname-a con 0x7f4430141f90 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4430139170 0x7f4430131480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 10 monclient(hunting): picked mon.noname-b con 0x7f4430139170 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4430139cb0 con 0x7f4430141f90 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44300b0b70 con 0x7f4430139170 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443013b920 con 0x7f44300a8910 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.774+0000 7f443877c640 10 monclient(hunting): authenticate will time out at 2774.091285s 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.775+0000 7f4435d17640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4430141f90 0x7f443013b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.775+0000 7f4435d17640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4430141f90 0x7f443013b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41942/0 (socket says 172.21.15.69:41942) 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.775+0000 7f4435d17640 1 -- 172.21.15.69:0/4192914148 learned_addr learned my addr 172.21.15.69:0/4192914148 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.775+0000 7f4436d19640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4430139170 0x7f4430131480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.775+0000 7f4435d17640 10 monclient(hunting): get_auth_request con 0x7f4430141f90 auth_method 0 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.775+0000 7f4435d17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.775+0000 7f4435d17640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.775+0000 7f4435d17640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436d19640 10 monclient(hunting): get_auth_request con 0x7f4430139170 auth_method 0 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436d19640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436d19640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4435d17640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4435d17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4435d17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436518640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a8910 0x7f4430138600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436d19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436d19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:29.783 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436d19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:29.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436518640 10 monclient(hunting): get_auth_request con 0x7f44300a8910 auth_method 0 2023-12-15T17:24:29.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436518640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:29.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436518640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:29.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.776+0000 7f4436518640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:29.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.777+0000 7f4436518640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:29.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.777+0000 7f4436518640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:29.784 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.777+0000 7f4436518640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:29.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.779+0000 7f4435d17640 10 monclient(hunting): handle_auth_done global_id 16146 payload 824 2023-12-15T17:24:29.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.779+0000 7f4435d17640 1 -- 172.21.15.69:0/4192914148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a8910 msgr2=0x7f4430138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:29.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.779+0000 7f4435d17640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a8910 0x7f4430138600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:29.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.779+0000 7f4435d17640 1 -- 172.21.15.69:0/4192914148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4430139170 msgr2=0x7f4430131480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:29.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.779+0000 7f4435d17640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4430139170 0x7f4430131480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:29.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.779+0000 7f4435d17640 10 monclient: _finish_hunting 0 2023-12-15T17:24:29.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.779+0000 7f4435d17640 1 monclient: found mon.noname-a 2023-12-15T17:24:29.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.779+0000 7f4435d17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:29.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.779+0000 7f4435d17640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-12-15T17:24:29.785 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.779+0000 7f4435d17640 1 -- 172.21.15.69:0/4192914148 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44200094d0 con 0x7f4430141f90 2023-12-15T17:24:29.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.781+0000 7f4435d17640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4430141f90 0x7f443013b3e0 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f4420000c00 tx=0x7f4420004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:29.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.781+0000 7f4426ffd640 10 client.?.objecter ms_handle_connect 0x7f4430141f90 2023-12-15T17:24:29.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.781+0000 7f4426ffd640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 1 -- 172.21.15.69:0/4192914148 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420039c90 con 0x7f4430141f90 2023-12-15T17:24:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-15T17:24:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 10 monclient: dump: 2023-12-15T17:24:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 10 monclient: _finish_auth 0 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 1 -- 172.21.15.69:0/4192914148 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4420039e10 con 0x7f4430141f90 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 1 -- 172.21.15.69:0/4192914148 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420043640 con 0x7f4430141f90 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4426ffd640 10 monclient: dump: 2023-12-15T17:24:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f443877c640 5 monclient: authenticate success, global_id 16146 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f443877c640 10 monclient: _renew_subs 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f443877c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f443877c640 1 -- 172.21.15.69:0/4192914148 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4430137aa0 con 0x7f4430141f90 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f4434d15640 4 set_mon_vals no callback set 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f443877c640 10 client.16146.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f443877c640 10 monclient: _renew_subs 2023-12-15T17:24:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f443877c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:24:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f443877c640 1 -- 172.21.15.69:0/4192914148 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4430139840 con 0x7f4430141f90 2023-12-15T17:24:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.786+0000 7f443877c640 1 librados: init done 2023-12-15T17:24:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.787+0000 7f443877c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.787+0000 7f443877c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.787+0000 7f443877c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-12-15T17:24:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:29.787+0000 7f443877c640 1 -- 172.21.15.69:0/4192914148 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4430139df0 con 0x7f4430141f90 2023-12-15T17:24:30.034 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.033+0000 7f4426ffd640 1 -- 172.21.15.69:0/4192914148 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f44200437a0 con 0x7f4430141f90 2023-12-15T17:24:30.034 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.033+0000 7f4426ffd640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f441406cba0 0x7f441406f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:30.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.033+0000 7f4436518640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f441406cba0 0x7f441406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:30.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.033+0000 7f4436518640 10 monclient: get_auth_request con 0x7f441406cba0 auth_method 0 2023-12-15T17:24:30.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.034+0000 7f4426ffd640 10 client.16146.objecter ms_handle_connect 0x7f441406cba0 2023-12-15T17:24:30.035 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.034+0000 7f4436518640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f441406cba0 0x7f441406f060 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f442c001780 tx=0x7f442c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:30.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.046+0000 7f4426ffd640 1 -- 172.21.15.69:0/4192914148 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7f442003e070 con 0x7f4430141f90 2023-12-15T17:24:30.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.046+0000 7f4426ffd640 10 client.16146.objecter ms_dispatch 0x7f4430000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:30.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.046+0000 7f4426ffd640 3 client.16146.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:30.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.046+0000 7f4426ffd640 3 client.16146.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:30.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.046+0000 7f4426ffd640 20 client.16146.objecter dump_active .. 0 homeless 2023-12-15T17:24:30.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.046+0000 7f4426ffd640 1 -- 172.21.15.69:0/4192914148 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f4420077f90 con 0x7f4430141f90 2023-12-15T17:24:30.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.046+0000 7f4426ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:30.048 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.046+0000 7f4426ffd640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:30.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.217+0000 7f443877c640 1 -- 172.21.15.69:0/4192914148 --> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4430139df0 con 0x7f441406cba0 2023-12-15T17:24:30.224 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.222+0000 7f4426ffd640 1 -- 172.21.15.69:0/4192914148 <== mgr.4107 v2:172.21.15.192:6800/4171480121 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152452 (secure 0 0 0) 0x7f4430139df0 con 0x7f441406cba0 2023-12-15T17:24:30.225 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-15T17:24:30.226 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-12-15T17:24:30.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 10 librados: watch_flush enter 2023-12-15T17:24:30.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 10 librados: watch_flush exit 2023-12-15T17:24:30.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 20 client.16146.objecter shutdown clearing up homeless session... 2023-12-15T17:24:30.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 10 client.16146.objecter successfully canceled tick 2023-12-15T17:24:30.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 -- 172.21.15.69:0/4192914148 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f441406cba0 msgr2=0x7f441406f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:30.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f441406cba0 0x7f441406f060 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f442c001780 tx=0x7f442c009040 comp rx=0 tx=0).stop 2023-12-15T17:24:30.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 10 monclient: shutdown 2023-12-15T17:24:30.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:30.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 -- 172.21.15.69:0/4192914148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4430141f90 msgr2=0x7f443013b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:30.228 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4430141f90 0x7f443013b3e0 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f4420000c00 tx=0x7f4420004730 comp rx=0 tx=0).stop 2023-12-15T17:24:30.229 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 -- 172.21.15.69:0/4192914148 shutdown_connections 2023-12-15T17:24:30.229 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f441406cba0 0x7f441406f060 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:30.229 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4430139170 0x7f4430131480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:30.229 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4430141f90 0x7f443013b3e0 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:30.229 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 --2- 172.21.15.69:0/4192914148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44300a8910 0x7f4430138600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:30.229 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 -- 172.21.15.69:0/4192914148 >> 172.21.15.69:0/4192914148 conn(0x7f44300b7780 msgr2=0x7f443013f180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:30.229 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 -- 172.21.15.69:0/4192914148 shutdown_connections 2023-12-15T17:24:30.229 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 -- 172.21.15.69:0/4192914148 wait complete. 2023-12-15T17:24:30.229 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.225+0000 7f443877c640 1 librados: shutdown 2023-12-15T17:24:30.238 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2023-12-15T17:24:29.787874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":570680,"kb_used_data":45048,"kb_used_omap":12,"kb_used_meta":525555,"kb_avail":754404040,"statfs":{"total":773094113280,"available":772509736960,"internally_reserved":0,"allocated":46129152,"data_stored":24842482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":538168880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002544"},"pg_stats":[{"pgid":"7.1b","version":"0'0","reported_seq":879,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:17.127362+0000","last_change":"2023-12-15T17:24:17.128020+0000","last_active":"2023-12-15T17:24:17.127362+0000","last_peered":"2023-12-15T17:24:17.127362+0000","last_clean":"2023-12-15T17:24:17.127362+0000","last_became_active":"2023-12-15T17:03:26.401879+0000","last_became_peered":"2023-12-15T17:03:26.401879+0000","last_unstale":"2023-12-15T17:24:17.127362+0000","last_undegraded":"2023-12-15T17:24:17.127362+0000","last_fullsized":"2023-12-15T17:24:17.127362+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:17.125956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:17.125956+0000","last_clean_scrub_stamp":"2023-12-15T17:24:17.125956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:26:51.664635+0000","scrub_duration":0.065328339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":887,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:22.467393+0000","last_change":"2023-12-15T17:24:22.468079+0000","last_active":"2023-12-15T17:24:22.467393+0000","last_peered":"2023-12-15T17:24:22.467393+0000","last_clean":"2023-12-15T17:24:22.467393+0000","last_became_active":"2023-12-15T17:03:43.159317+0000","last_became_peered":"2023-12-15T17:03:43.159317+0000","last_unstale":"2023-12-15T17:24:22.467393+0000","last_undegraded":"2023-12-15T17:24:22.467393+0000","last_fullsized":"2023-12-15T17:24:22.467393+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:22.466023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:22.466023+0000","last_clean_scrub_stamp":"2023-12-15T17:24:22.466023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:42:49.963637+0000","scrub_duration":0.031272869000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":887,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:23.491155+0000","last_change":"2023-12-15T17:24:23.491804+0000","last_active":"2023-12-15T17:24:23.491155+0000","last_peered":"2023-12-15T17:24:23.491155+0000","last_clean":"2023-12-15T17:24:23.491155+0000","last_became_active":"2023-12-15T17:03:27.137993+0000","last_became_peered":"2023-12-15T17:03:27.137993+0000","last_unstale":"2023-12-15T17:24:23.491155+0000","last_undegraded":"2023-12-15T17:24:23.491155+0000","last_fullsized":"2023-12-15T17:24:23.491155+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:23.489841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:23.489841+0000","last_clean_scrub_stamp":"2023-12-15T17:24:23.489841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T20:17:20.772223+0000","scrub_duration":0.039902588000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":879,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:19.103386+0000","last_change":"2023-12-15T17:24:19.104037+0000","last_active":"2023-12-15T17:24:19.103386+0000","last_peered":"2023-12-15T17:24:19.103386+0000","last_clean":"2023-12-15T17:24:19.103386+0000","last_became_active":"2023-12-15T17:03:32.155432+0000","last_became_peered":"2023-12-15T17:03:32.155432+0000","last_unstale":"2023-12-15T17:24:19.103386+0000","last_undegraded":"2023-12-15T17:24:19.103386+0000","last_fullsized":"2023-12-15T17:24:19.103386+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:19.101729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:19.101729+0000","last_clean_scrub_stamp":"2023-12-15T17:24:19.101729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:03:26.019169+0000","scrub_duration":0.026386456999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":857,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:03.959482+0000","last_change":"2023-12-15T17:24:03.959482+0000","last_active":"2023-12-15T17:24:03.959482+0000","last_peered":"2023-12-15T17:24:03.959482+0000","last_clean":"2023-12-15T17:24:03.959482+0000","last_became_active":"2023-12-15T17:03:32.040162+0000","last_became_peered":"2023-12-15T17:03:32.040162+0000","last_unstale":"2023-12-15T17:24:03.959482+0000","last_undegraded":"2023-12-15T17:24:03.959482+0000","last_fullsized":"2023-12-15T17:24:03.959482+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:03.957092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:03.957092+0000","last_clean_scrub_stamp":"2023-12-15T17:24:03.957092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:21:14.132235+0000","scrub_duration":0.071783557999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":864,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:24.445282+0000","last_change":"2023-12-15T17:24:24.445959+0000","last_active":"2023-12-15T17:24:24.445282+0000","last_peered":"2023-12-15T17:24:24.445282+0000","last_clean":"2023-12-15T17:24:24.445282+0000","last_became_active":"2023-12-15T17:03:29.584729+0000","last_became_peered":"2023-12-15T17:03:29.584729+0000","last_unstale":"2023-12-15T17:24:24.445282+0000","last_undegraded":"2023-12-15T17:24:24.445282+0000","last_fullsized":"2023-12-15T17:24:24.445282+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:24.443814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:24.443814+0000","last_clean_scrub_stamp":"2023-12-15T17:24:24.443814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:41:40.347880+0000","scrub_duration":0.053882645999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":882,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:12.743325+0000","last_change":"2023-12-15T17:24:12.743971+0000","last_active":"2023-12-15T17:24:12.743325+0000","last_peered":"2023-12-15T17:24:12.743325+0000","last_clean":"2023-12-15T17:24:12.743325+0000","last_became_active":"2023-12-15T17:03:27.349390+0000","last_became_peered":"2023-12-15T17:03:27.349390+0000","last_unstale":"2023-12-15T17:24:12.743325+0000","last_undegraded":"2023-12-15T17:24:12.743325+0000","last_fullsized":"2023-12-15T17:24:12.743325+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:12.742035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:12.742035+0000","last_clean_scrub_stamp":"2023-12-15T17:24:12.742035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T23:07:28.945880+0000","scrub_duration":0.065679818000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":859,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:09.575427+0000","last_change":"2023-12-15T17:24:09.576130+0000","last_active":"2023-12-15T17:24:09.575427+0000","last_peered":"2023-12-15T17:24:09.575427+0000","last_clean":"2023-12-15T17:24:09.575427+0000","last_became_active":"2023-12-15T17:03:41.182883+0000","last_became_peered":"2023-12-15T17:03:41.182883+0000","last_unstale":"2023-12-15T17:24:09.575427+0000","last_undegraded":"2023-12-15T17:24:09.575427+0000","last_fullsized":"2023-12-15T17:24:09.575427+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:09.573650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:09.573650+0000","last_clean_scrub_stamp":"2023-12-15T17:24:09.573650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:45:36.700448+0000","scrub_duration":0.028287306000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":862,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:11.446294+0000","last_change":"2023-12-15T17:24:11.446985+0000","last_active":"2023-12-15T17:24:11.446294+0000","last_peered":"2023-12-15T17:24:11.446294+0000","last_clean":"2023-12-15T17:24:11.446294+0000","last_became_active":"2023-12-15T17:03:27.563090+0000","last_became_peered":"2023-12-15T17:03:27.563090+0000","last_unstale":"2023-12-15T17:24:11.446294+0000","last_undegraded":"2023-12-15T17:24:11.446294+0000","last_fullsized":"2023-12-15T17:24:11.446294+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:11.444844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:11.444844+0000","last_clean_scrub_stamp":"2023-12-15T17:24:11.444844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T00:32:05.642935+0000","scrub_duration":0.066389853999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":858,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:06.795812+0000","last_change":"2023-12-15T17:24:06.795812+0000","last_active":"2023-12-15T17:24:06.795812+0000","last_peered":"2023-12-15T17:24:06.795812+0000","last_clean":"2023-12-15T17:24:06.795812+0000","last_became_active":"2023-12-15T17:03:26.168812+0000","last_became_peered":"2023-12-15T17:03:26.168812+0000","last_unstale":"2023-12-15T17:24:06.795812+0000","last_undegraded":"2023-12-15T17:24:06.795812+0000","last_fullsized":"2023-12-15T17:24:06.795812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:06.793573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:06.793573+0000","last_clean_scrub_stamp":"2023-12-15T17:24:06.793573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:32:36.894303+0000","scrub_duration":0.032537192999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":860,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:05.659597+0000","last_change":"2023-12-15T17:24:05.659597+0000","last_active":"2023-12-15T17:24:05.659597+0000","last_peered":"2023-12-15T17:24:05.659597+0000","last_clean":"2023-12-15T17:24:05.659597+0000","last_became_active":"2023-12-15T17:03:33.660278+0000","last_became_peered":"2023-12-15T17:03:33.660278+0000","last_unstale":"2023-12-15T17:24:05.659597+0000","last_undegraded":"2023-12-15T17:24:05.659597+0000","last_fullsized":"2023-12-15T17:24:05.659597+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:05.657539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:05.657539+0000","last_clean_scrub_stamp":"2023-12-15T17:24:05.657539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:13:03.299750+0000","scrub_duration":0.038816977000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":863,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:14.388837+0000","last_change":"2023-12-15T17:24:14.388837+0000","last_active":"2023-12-15T17:24:14.388837+0000","last_peered":"2023-12-15T17:24:14.388837+0000","last_clean":"2023-12-15T17:24:14.388837+0000","last_became_active":"2023-12-15T17:03:29.479817+0000","last_became_peered":"2023-12-15T17:03:29.479817+0000","last_unstale":"2023-12-15T17:24:14.388837+0000","last_undegraded":"2023-12-15T17:24:14.388837+0000","last_fullsized":"2023-12-15T17:24:14.388837+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:14.386665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:14.386665+0000","last_clean_scrub_stamp":"2023-12-15T17:24:14.386665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:41:02.726380+0000","scrub_duration":0.03370915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":881,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:12.157934+0000","last_change":"2023-12-15T17:24:12.157934+0000","last_active":"2023-12-15T17:24:12.157934+0000","last_peered":"2023-12-15T17:24:12.157934+0000","last_clean":"2023-12-15T17:24:12.157934+0000","last_became_active":"2023-12-15T17:03:27.764430+0000","last_became_peered":"2023-12-15T17:03:27.764430+0000","last_unstale":"2023-12-15T17:24:12.157934+0000","last_undegraded":"2023-12-15T17:24:12.157934+0000","last_fullsized":"2023-12-15T17:24:12.157934+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:12.155519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:12.155519+0000","last_clean_scrub_stamp":"2023-12-15T17:24:12.155519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T00:05:00.046940+0000","scrub_duration":0.028919402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":893,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:12.394416+0000","last_change":"2023-12-15T17:24:12.395066+0000","last_active":"2023-12-15T17:24:12.394416+0000","last_peered":"2023-12-15T17:24:12.394416+0000","last_clean":"2023-12-15T17:24:12.394416+0000","last_became_active":"2023-12-15T17:02:01.275938+0000","last_became_peered":"2023-12-15T17:02:01.275938+0000","last_unstale":"2023-12-15T17:24:12.394416+0000","last_undegraded":"2023-12-15T17:24:12.394416+0000","last_fullsized":"2023-12-15T17:24:12.394416+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:12.393022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:12.393022+0000","last_clean_scrub_stamp":"2023-12-15T17:24:12.393022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:09:59.895953+0000","scrub_duration":0.025036952000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":884,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:11.742695+0000","last_change":"2023-12-15T17:24:11.742695+0000","last_active":"2023-12-15T17:24:11.742695+0000","last_peered":"2023-12-15T17:24:11.742695+0000","last_clean":"2023-12-15T17:24:11.742695+0000","last_became_active":"2023-12-15T17:03:29.736171+0000","last_became_peered":"2023-12-15T17:03:29.736171+0000","last_unstale":"2023-12-15T17:24:11.742695+0000","last_undegraded":"2023-12-15T17:24:11.742695+0000","last_fullsized":"2023-12-15T17:24:11.742695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:11.740811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:11.740811+0000","last_clean_scrub_stamp":"2023-12-15T17:24:11.740811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:26:23.065300+0000","scrub_duration":0.059800056999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":886,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:11.128826+0000","last_change":"2023-12-15T17:24:11.129519+0000","last_active":"2023-12-15T17:24:11.128826+0000","last_peered":"2023-12-15T17:24:11.128826+0000","last_clean":"2023-12-15T17:24:11.128826+0000","last_became_active":"2023-12-15T17:01:59.918026+0000","last_became_peered":"2023-12-15T17:01:59.918026+0000","last_unstale":"2023-12-15T17:24:11.128826+0000","last_undegraded":"2023-12-15T17:24:11.128826+0000","last_fullsized":"2023-12-15T17:24:11.128826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:11.127378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:11.127378+0000","last_clean_scrub_stamp":"2023-12-15T17:24:11.127378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T00:16:47.199468+0000","scrub_duration":0.024460342,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":859,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:10.850772+0000","last_change":"2023-12-15T17:24:10.851410+0000","last_active":"2023-12-15T17:24:10.850772+0000","last_peered":"2023-12-15T17:24:10.850772+0000","last_clean":"2023-12-15T17:24:10.850772+0000","last_became_active":"2023-12-15T17:03:26.235756+0000","last_became_peered":"2023-12-15T17:03:26.235756+0000","last_unstale":"2023-12-15T17:24:10.850772+0000","last_undegraded":"2023-12-15T17:24:10.850772+0000","last_fullsized":"2023-12-15T17:24:10.850772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:10.849475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:10.849475+0000","last_clean_scrub_stamp":"2023-12-15T17:24:10.849475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T02:27:40.837006+0000","scrub_duration":0.021818987000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":893,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:19.429813+0000","last_change":"2023-12-15T17:24:19.430870+0000","last_active":"2023-12-15T17:24:19.429813+0000","last_peered":"2023-12-15T17:24:19.429813+0000","last_clean":"2023-12-15T17:24:19.429813+0000","last_became_active":"2023-12-15T17:02:01.375672+0000","last_became_peered":"2023-12-15T17:02:01.375672+0000","last_unstale":"2023-12-15T17:24:19.429813+0000","last_undegraded":"2023-12-15T17:24:19.429813+0000","last_fullsized":"2023-12-15T17:24:19.429813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:19.428306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:19.428306+0000","last_clean_scrub_stamp":"2023-12-15T17:24:19.428306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:36:49.144604+0000","scrub_duration":0.025855765999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"908'8","reported_seq":875,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:07.695354+0000","last_change":"2023-12-15T17:24:07.695354+0000","last_active":"2023-12-15T17:24:07.695354+0000","last_peered":"2023-12-15T17:24:07.695354+0000","last_clean":"2023-12-15T17:24:07.695354+0000","last_became_active":"2023-12-15T17:03:26.343248+0000","last_became_peered":"2023-12-15T17:03:26.343248+0000","last_unstale":"2023-12-15T17:24:07.695354+0000","last_undegraded":"2023-12-15T17:24:07.695354+0000","last_fullsized":"2023-12-15T17:24:07.695354+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"908'8","last_scrub_stamp":"2023-12-15T17:24:07.692843+0000","last_deep_scrub":"908'8","last_deep_scrub_stamp":"2023-12-15T17:24:07.692843+0000","last_clean_scrub_stamp":"2023-12-15T17:24:07.692843+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:17:15.243313+0000","scrub_duration":0.096291876999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":890,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:08.829167+0000","last_change":"2023-12-15T17:24:08.833619+0000","last_active":"2023-12-15T17:24:08.829167+0000","last_peered":"2023-12-15T17:24:08.829167+0000","last_clean":"2023-12-15T17:24:08.829167+0000","last_became_active":"2023-12-15T17:01:59.908680+0000","last_became_peered":"2023-12-15T17:01:59.908680+0000","last_unstale":"2023-12-15T17:24:08.829167+0000","last_undegraded":"2023-12-15T17:24:08.829167+0000","last_fullsized":"2023-12-15T17:24:08.829167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:08.825473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:08.825473+0000","last_clean_scrub_stamp":"2023-12-15T17:24:08.825473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:44:54.082682+0000","scrub_duration":0.18921722799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":887,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:27.610087+0000","last_change":"2023-12-15T17:24:27.610773+0000","last_active":"2023-12-15T17:24:27.610087+0000","last_peered":"2023-12-15T17:24:27.610087+0000","last_clean":"2023-12-15T17:24:27.610087+0000","last_became_active":"2023-12-15T17:03:27.780434+0000","last_became_peered":"2023-12-15T17:03:27.780434+0000","last_unstale":"2023-12-15T17:24:27.610087+0000","last_undegraded":"2023-12-15T17:24:27.610087+0000","last_fullsized":"2023-12-15T17:24:27.610087+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:27.608671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:27.608671+0000","last_clean_scrub_stamp":"2023-12-15T17:24:27.608671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:31:34.816556+0000","scrub_duration":0.065624474000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":894,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:08.349739+0000","last_change":"2023-12-15T17:24:08.350421+0000","last_active":"2023-12-15T17:24:08.349739+0000","last_peered":"2023-12-15T17:24:08.349739+0000","last_clean":"2023-12-15T17:24:08.349739+0000","last_became_active":"2023-12-15T17:02:01.713188+0000","last_became_peered":"2023-12-15T17:02:01.713188+0000","last_unstale":"2023-12-15T17:24:08.349739+0000","last_undegraded":"2023-12-15T17:24:08.349739+0000","last_fullsized":"2023-12-15T17:24:08.349739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:08.348302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:08.348302+0000","last_clean_scrub_stamp":"2023-12-15T17:24:08.348302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T20:49:09.464728+0000","scrub_duration":0.026008239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":881,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:18.073915+0000","last_change":"2023-12-15T17:24:18.073915+0000","last_active":"2023-12-15T17:24:18.073915+0000","last_peered":"2023-12-15T17:24:18.073915+0000","last_clean":"2023-12-15T17:24:18.073915+0000","last_became_active":"2023-12-15T17:03:26.252650+0000","last_became_peered":"2023-12-15T17:03:26.252650+0000","last_unstale":"2023-12-15T17:24:18.073915+0000","last_undegraded":"2023-12-15T17:24:18.073915+0000","last_fullsized":"2023-12-15T17:24:18.073915+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:18.071889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:18.071889+0000","last_clean_scrub_stamp":"2023-12-15T17:24:18.071889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T00:24:25.706185+0000","scrub_duration":0.028403799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":869,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:09.352946+0000","last_change":"2023-12-15T17:24:09.353649+0000","last_active":"2023-12-15T17:24:09.352946+0000","last_peered":"2023-12-15T17:24:09.352946+0000","last_clean":"2023-12-15T17:24:09.352946+0000","last_became_active":"2023-12-15T17:01:59.531480+0000","last_became_peered":"2023-12-15T17:01:59.531480+0000","last_unstale":"2023-12-15T17:24:09.352946+0000","last_undegraded":"2023-12-15T17:24:09.352946+0000","last_fullsized":"2023-12-15T17:24:09.352946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:09.351490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:09.351490+0000","last_clean_scrub_stamp":"2023-12-15T17:24:09.351490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T05:10:50.376906+0000","scrub_duration":0.023450242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":888,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:26.536041+0000","last_change":"2023-12-15T17:24:26.536734+0000","last_active":"2023-12-15T17:24:26.536041+0000","last_peered":"2023-12-15T17:24:26.536041+0000","last_clean":"2023-12-15T17:24:26.536041+0000","last_became_active":"2023-12-15T17:03:29.361304+0000","last_became_peered":"2023-12-15T17:03:29.361304+0000","last_unstale":"2023-12-15T17:24:26.536041+0000","last_undegraded":"2023-12-15T17:24:26.536041+0000","last_fullsized":"2023-12-15T17:24:26.536041+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:26.530131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:26.530131+0000","last_clean_scrub_stamp":"2023-12-15T17:24:26.530131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:18:53.298936+0000","scrub_duration":0.037571341000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":894,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:07.523169+0000","last_change":"2023-12-15T17:24:07.523169+0000","last_active":"2023-12-15T17:24:07.523169+0000","last_peered":"2023-12-15T17:24:07.523169+0000","last_clean":"2023-12-15T17:24:07.523169+0000","last_became_active":"2023-12-15T17:02:01.208353+0000","last_became_peered":"2023-12-15T17:02:01.208353+0000","last_unstale":"2023-12-15T17:24:07.523169+0000","last_undegraded":"2023-12-15T17:24:07.523169+0000","last_fullsized":"2023-12-15T17:24:07.523169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:07.514766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:07.514766+0000","last_clean_scrub_stamp":"2023-12-15T17:24:07.514766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:25:52.283874+0000","scrub_duration":0.192829009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":883,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:18.740173+0000","last_change":"2023-12-15T17:24:18.740810+0000","last_active":"2023-12-15T17:24:18.740173+0000","last_peered":"2023-12-15T17:24:18.740173+0000","last_clean":"2023-12-15T17:24:18.740173+0000","last_became_active":"2023-12-15T17:03:27.321338+0000","last_became_peered":"2023-12-15T17:03:27.321338+0000","last_unstale":"2023-12-15T17:24:18.740173+0000","last_undegraded":"2023-12-15T17:24:18.740173+0000","last_fullsized":"2023-12-15T17:24:18.740173+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:18.738864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:18.738864+0000","last_clean_scrub_stamp":"2023-12-15T17:24:18.738864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:15:04.578031+0000","scrub_duration":0.028733149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":894,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:13.418393+0000","last_change":"2023-12-15T17:24:13.419038+0000","last_active":"2023-12-15T17:24:13.418393+0000","last_peered":"2023-12-15T17:24:13.418393+0000","last_clean":"2023-12-15T17:24:13.418393+0000","last_became_active":"2023-12-15T17:02:01.616517+0000","last_became_peered":"2023-12-15T17:02:01.616517+0000","last_unstale":"2023-12-15T17:24:13.418393+0000","last_undegraded":"2023-12-15T17:24:13.418393+0000","last_fullsized":"2023-12-15T17:24:13.418393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:13.417057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:13.417057+0000","last_clean_scrub_stamp":"2023-12-15T17:24:13.417057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:02:05.953533+0000","scrub_duration":0.022183317000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"908'78","reported_seq":974,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:04.888918+0000","last_change":"2023-12-15T17:24:04.890040+0000","last_active":"2023-12-15T17:24:04.888918+0000","last_peered":"2023-12-15T17:24:04.888918+0000","last_clean":"2023-12-15T17:24:04.888918+0000","last_became_active":"2023-12-15T17:01:52.878062+0000","last_became_peered":"2023-12-15T17:01:52.878062+0000","last_unstale":"2023-12-15T17:24:04.888918+0000","last_undegraded":"2023-12-15T17:24:04.888918+0000","last_fullsized":"2023-12-15T17:24:04.888918+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"908'78","last_scrub_stamp":"2023-12-15T17:24:04.887529+0000","last_deep_scrub":"908'78","last_deep_scrub_stamp":"2023-12-15T17:24:04.887529+0000","last_clean_scrub_stamp":"2023-12-15T17:24:04.887529+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:14:33.227165+0000","scrub_duration":0.131867237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":860,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:17.557346+0000","last_change":"2023-12-15T17:24:17.557991+0000","last_active":"2023-12-15T17:24:17.557346+0000","last_peered":"2023-12-15T17:24:17.557346+0000","last_clean":"2023-12-15T17:24:17.557346+0000","last_became_active":"2023-12-15T17:03:31.075388+0000","last_became_peered":"2023-12-15T17:03:31.075388+0000","last_unstale":"2023-12-15T17:24:17.557346+0000","last_undegraded":"2023-12-15T17:24:17.557346+0000","last_fullsized":"2023-12-15T17:24:17.557346+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:17.556069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:17.556069+0000","last_clean_scrub_stamp":"2023-12-15T17:24:17.556069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T22:34:14.282587+0000","scrub_duration":0.026824982000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":858,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:14.957800+0000","last_change":"2023-12-15T17:24:14.958448+0000","last_active":"2023-12-15T17:24:14.957800+0000","last_peered":"2023-12-15T17:24:14.957800+0000","last_clean":"2023-12-15T17:24:14.957800+0000","last_became_active":"2023-12-15T17:03:29.683464+0000","last_became_peered":"2023-12-15T17:03:29.683464+0000","last_unstale":"2023-12-15T17:24:14.957800+0000","last_undegraded":"2023-12-15T17:24:14.957800+0000","last_fullsized":"2023-12-15T17:24:14.957800+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:14.956518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:14.956518+0000","last_clean_scrub_stamp":"2023-12-15T17:24:14.956518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:18:51.569911+0000","scrub_duration":0.054306822999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":863,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:26.461905+0000","last_change":"2023-12-15T17:24:26.462644+0000","last_active":"2023-12-15T17:24:26.461905+0000","last_peered":"2023-12-15T17:24:26.461905+0000","last_clean":"2023-12-15T17:24:26.461905+0000","last_became_active":"2023-12-15T17:03:26.036270+0000","last_became_peered":"2023-12-15T17:03:26.036270+0000","last_unstale":"2023-12-15T17:24:26.461905+0000","last_undegraded":"2023-12-15T17:24:26.461905+0000","last_fullsized":"2023-12-15T17:24:26.461905+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:26.460433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:26.460433+0000","last_clean_scrub_stamp":"2023-12-15T17:24:26.460433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T00:00:07.891658+0000","scrub_duration":0.039012049,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":861,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:16.452378+0000","last_change":"2023-12-15T17:24:16.453016+0000","last_active":"2023-12-15T17:24:16.452378+0000","last_peered":"2023-12-15T17:24:16.452378+0000","last_clean":"2023-12-15T17:24:16.452378+0000","last_became_active":"2023-12-15T17:03:29.608805+0000","last_became_peered":"2023-12-15T17:03:29.608805+0000","last_unstale":"2023-12-15T17:24:16.452378+0000","last_undegraded":"2023-12-15T17:24:16.452378+0000","last_fullsized":"2023-12-15T17:24:16.452378+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:16.451110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:16.451110+0000","last_clean_scrub_stamp":"2023-12-15T17:24:16.451110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T18:26:15.281353+0000","scrub_duration":0.064177177000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":884,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:20.786160+0000","last_change":"2023-12-15T17:24:20.786821+0000","last_active":"2023-12-15T17:24:20.786160+0000","last_peered":"2023-12-15T17:24:20.786160+0000","last_clean":"2023-12-15T17:24:20.786160+0000","last_became_active":"2023-12-15T17:03:27.499200+0000","last_became_peered":"2023-12-15T17:03:27.499200+0000","last_unstale":"2023-12-15T17:24:20.786160+0000","last_undegraded":"2023-12-15T17:24:20.786160+0000","last_fullsized":"2023-12-15T17:24:20.786160+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:20.784814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:20.784814+0000","last_clean_scrub_stamp":"2023-12-15T17:24:20.784814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T21:38:08.427854+0000","scrub_duration":0.021405158000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":884,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:09.684414+0000","last_change":"2023-12-15T17:24:09.685061+0000","last_active":"2023-12-15T17:24:09.684414+0000","last_peered":"2023-12-15T17:24:09.684414+0000","last_clean":"2023-12-15T17:24:09.684414+0000","last_became_active":"2023-12-15T17:03:28.012736+0000","last_became_peered":"2023-12-15T17:03:28.012736+0000","last_unstale":"2023-12-15T17:24:09.684414+0000","last_undegraded":"2023-12-15T17:24:09.684414+0000","last_fullsized":"2023-12-15T17:24:09.684414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:09.683099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:09.683099+0000","last_clean_scrub_stamp":"2023-12-15T17:24:09.683099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T17:32:24.162433+0000","scrub_duration":0.029324424000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":880,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:14.140623+0000","last_change":"2023-12-15T17:24:14.141340+0000","last_active":"2023-12-15T17:24:14.140623+0000","last_peered":"2023-12-15T17:24:14.140623+0000","last_clean":"2023-12-15T17:24:14.140623+0000","last_became_active":"2023-12-15T17:03:27.791224+0000","last_became_peered":"2023-12-15T17:03:27.791224+0000","last_unstale":"2023-12-15T17:24:14.140623+0000","last_undegraded":"2023-12-15T17:24:14.140623+0000","last_fullsized":"2023-12-15T17:24:14.140623+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:14.139233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:14.139233+0000","last_clean_scrub_stamp":"2023-12-15T17:24:14.139233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T03:31:59.142347+0000","scrub_duration":0.037614269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":860,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:15.370968+0000","last_change":"2023-12-15T17:24:15.371623+0000","last_active":"2023-12-15T17:24:15.370968+0000","last_peered":"2023-12-15T17:24:15.370968+0000","last_clean":"2023-12-15T17:24:15.370968+0000","last_became_active":"2023-12-15T17:03:33.897262+0000","last_became_peered":"2023-12-15T17:03:33.897262+0000","last_unstale":"2023-12-15T17:24:15.370968+0000","last_undegraded":"2023-12-15T17:24:15.370968+0000","last_fullsized":"2023-12-15T17:24:15.370968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:15.369678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:15.369678+0000","last_clean_scrub_stamp":"2023-12-15T17:24:15.369678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:50:07.952142+0000","scrub_duration":0.051403880999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":884,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:17.743217+0000","last_change":"2023-12-15T17:24:17.743217+0000","last_active":"2023-12-15T17:24:17.743217+0000","last_peered":"2023-12-15T17:24:17.743217+0000","last_clean":"2023-12-15T17:24:17.743217+0000","last_became_active":"2023-12-15T17:03:28.164968+0000","last_became_peered":"2023-12-15T17:03:28.164968+0000","last_unstale":"2023-12-15T17:24:17.743217+0000","last_undegraded":"2023-12-15T17:24:17.743217+0000","last_fullsized":"2023-12-15T17:24:17.743217+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:17.741233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:17.741233+0000","last_clean_scrub_stamp":"2023-12-15T17:24:17.741233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T05:22:28.027364+0000","scrub_duration":0.025847054000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":886,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:20.425635+0000","last_change":"2023-12-15T17:24:20.426336+0000","last_active":"2023-12-15T17:24:20.425635+0000","last_peered":"2023-12-15T17:24:20.425635+0000","last_clean":"2023-12-15T17:24:20.425635+0000","last_became_active":"2023-12-15T17:03:29.219241+0000","last_became_peered":"2023-12-15T17:03:29.219241+0000","last_unstale":"2023-12-15T17:24:20.425635+0000","last_undegraded":"2023-12-15T17:24:20.425635+0000","last_fullsized":"2023-12-15T17:24:20.425635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:20.424210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:20.424210+0000","last_clean_scrub_stamp":"2023-12-15T17:24:20.424210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T04:37:47.808658+0000","scrub_duration":0.047814219999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":857,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:20.851810+0000","last_change":"2023-12-15T17:24:20.852480+0000","last_active":"2023-12-15T17:24:20.851810+0000","last_peered":"2023-12-15T17:24:20.851810+0000","last_clean":"2023-12-15T17:24:20.851810+0000","last_became_active":"2023-12-15T17:03:25.394333+0000","last_became_peered":"2023-12-15T17:03:25.394333+0000","last_unstale":"2023-12-15T17:24:20.851810+0000","last_undegraded":"2023-12-15T17:24:20.851810+0000","last_fullsized":"2023-12-15T17:24:20.851810+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:20.850453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:20.850453+0000","last_clean_scrub_stamp":"2023-12-15T17:24:20.850453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T01:00:13.109106+0000","scrub_duration":0.035800219000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":857,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-12-15T17:24:18.856706+0000","last_change":"2023-12-15T17:24:18.857394+0000","last_active":"2023-12-15T17:24:18.856706+0000","last_peered":"2023-12-15T17:24:18.856706+0000","last_clean":"2023-12-15T17:24:18.856706+0000","last_became_active":"2023-12-15T17:03:25.924138+0000","last_became_peered":"2023-12-15T17:03:25.924138+0000","last_unstale":"2023-12-15T17:24:18.856706+0000","last_undegraded":"2023-12-15T17:24:18.856706+0000","last_fullsized":"2023-12-15T17:24:18.856706+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T17:24:18.855323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T17:24:18.855323+0000","last_clean_scrub_stamp":"2023-12-15T17:24:18.855323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-16T19:31:05.112903+0000","scrub_duration":0.071391967000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58920,"kb_used_data":5536,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312920,"statfs":{"total":96636764160,"available":96576430080,"internally_reserved":0,"allocated":5668864,"data_stored":3005854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":10.289999999999999,"5min":13.026999999999999,"15min":16.065000000000001},"min":{"1min":5.0090000000000003,"5min":4.351,"15min":4.3390000000000004},"max":{"1min":20.521000000000001,"5min":151.61099999999999,"15min":199.32400000000001},"last":11.818},{"interface":"front","average":{"1min":10.064,"5min":14.347,"15min":17.850999999999999},"min":{"1min":6.2839999999999998,"5min":5.0149999999999997,"15min":4.9370000000000003},"max":{"1min":20.202999999999999,"5min":278.32799999999997,"15min":330.613},"last":7.569}]},{"osd":1,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":16.135000000000002,"5min":17.571999999999999,"15min":19.690000000000001},"min":{"1min":7.4400000000000004,"5min":6.7439999999999998,"15min":5.726},"max":{"1min":65.194999999999993,"5min":159.22399999999999,"15min":303.95400000000001},"last":9.7509999999999994},{"interface":"front","average":{"1min":15.039999999999999,"5min":19.803000000000001,"15min":20.053999999999998},"min":{"1min":7.5439999999999996,"5min":6.923,"15min":6.4610000000000003},"max":{"1min":55.744,"5min":326.125,"15min":326.125},"last":8.9879999999999995}]},{"osd":3,"last update":"Fri Dec 15 17:24:06 2023","interfaces":[{"interface":"back","average":{"1min":10.157999999999999,"5min":17.463000000000001,"15min":19.84},"min":{"1min":6.1559999999999997,"5min":6.1559999999999997,"15min":5.9050000000000002},"max":{"1min":21.367000000000001,"5min":309.05399999999997,"15min":309.05399999999997},"last":18.922000000000001},{"interface":"front","average":{"1min":16.939,"5min":21.664000000000001,"15min":22.506},"min":{"1min":9.5,"5min":7.2480000000000002,"15min":6.1449999999999996},"max":{"1min":27.216000000000001,"5min":372.85199999999998,"15min":372.85199999999998},"last":21.146000000000001}]},{"osd":4,"last update":"Fri Dec 15 17:23:44 2023","interfaces":[{"interface":"back","average":{"1min":15.255000000000001,"5min":23.908000000000001,"15min":24.216000000000001},"min":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":6.5359999999999996},"max":{"1min":26.652000000000001,"5min":397.04000000000002,"15min":397.04000000000002},"last":16.408000000000001},{"interface":"front","average":{"1min":16.533999999999999,"5min":24.268999999999998,"15min":25.311},"min":{"1min":8.5980000000000008,"5min":8.5980000000000008,"15min":7.7809999999999997},"max":{"1min":31.385999999999999,"5min":217.01599999999999,"15min":217.01599999999999},"last":12.574}]},{"osd":5,"last update":"Fri Dec 15 17:24:13 2023","interfaces":[{"interface":"back","average":{"1min":13.855,"5min":18.257000000000001,"15min":26.713000000000001},"min":{"1min":8.1959999999999997,"5min":8.1959999999999997,"15min":7.8440000000000003},"max":{"1min":25.966000000000001,"5min":91.251999999999995,"15min":290.19299999999998},"last":7.9139999999999997},{"interface":"front","average":{"1min":17.928999999999998,"5min":14.909000000000001,"15min":23.271999999999998},"min":{"1min":10.686999999999999,"5min":5.9160000000000004,"15min":5.9160000000000004},"max":{"1min":32.569000000000003,"5min":64.555999999999997,"15min":280.45400000000001},"last":10.884}]},{"osd":6,"last update":"Fri Dec 15 17:23:49 2023","interfaces":[{"interface":"back","average":{"1min":16.161000000000001,"5min":22.856000000000002,"15min":25.489999999999998},"min":{"1min":10.462,"5min":9.4039999999999999,"15min":8.5969999999999995},"max":{"1min":25.347999999999999,"5min":289.68400000000003,"15min":289.68400000000003},"last":18.212},{"interface":"front","average":{"1min":14.811999999999999,"5min":21.321999999999999,"15min":22.838999999999999},"min":{"1min":9.6989999999999998,"5min":7.8200000000000003,"15min":6.9409999999999998},"max":{"1min":22.036000000000001,"5min":274.80099999999999,"15min":274.80099999999999},"last":15.33}]},{"osd":7,"last update":"Fri Dec 15 17:24:06 2023","interfaces":[{"interface":"back","average":{"1min":20.538,"5min":24.663,"15min":26.561},"min":{"1min":11.538,"5min":9.3930000000000007,"15min":8.5820000000000007},"max":{"1min":46.052,"5min":278.98000000000002,"15min":278.98000000000002},"last":17.306000000000001},{"interface":"front","average":{"1min":17.202999999999999,"5min":22.379000000000001,"15min":24.699999999999999},"min":{"1min":10.885,"5min":7.8360000000000003,"15min":7.8360000000000003},"max":{"1min":28.640999999999998,"5min":275.72300000000001,"15min":275.72300000000001},"last":13.361000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509716,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81520,"kb_used_data":5928,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290320,"statfs":{"total":96636764160,"available":96553287680,"internally_reserved":0,"allocated":6070272,"data_stored":3403694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:24:17 2023","interfaces":[{"interface":"back","average":{"1min":11.086,"5min":15.568,"15min":18.765999999999998},"min":{"1min":4.9909999999999997,"5min":4.4870000000000001,"15min":4.383},"max":{"1min":34.609999999999999,"5min":94.881,"15min":295.94900000000001},"last":10.176},{"interface":"front","average":{"1min":12.356,"5min":13.305,"15min":15.557},"min":{"1min":5.6600000000000001,"5min":5.0419999999999998,"15min":4.3449999999999998},"max":{"1min":39.939,"5min":90.009,"15min":195.649},"last":16.285}]},{"osd":2,"last update":"Fri Dec 15 17:24:05 2023","interfaces":[{"interface":"back","average":{"1min":14.614000000000001,"5min":19.599,"15min":23.256},"min":{"1min":7.0810000000000004,"5min":5.7380000000000004,"15min":5.5190000000000001},"max":{"1min":95.677999999999997,"5min":276.47000000000003,"15min":482.72800000000001},"last":5.516},{"interface":"front","average":{"1min":15.545999999999999,"5min":22.382000000000001,"15min":28.507000000000001},"min":{"1min":6.8949999999999996,"5min":6.5629999999999997,"15min":6.1699999999999999},"max":{"1min":35.276000000000003,"5min":261.74400000000003,"15min":454.017},"last":11.349}]},{"osd":3,"last update":"Fri Dec 15 17:24:17 2023","interfaces":[{"interface":"back","average":{"1min":18.105,"5min":19.667999999999999,"15min":24.605},"min":{"1min":8.3040000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"max":{"1min":62.531999999999996,"5min":147.37100000000001,"15min":285.64499999999998},"last":10.779},{"interface":"front","average":{"1min":20.079000000000001,"5min":20.936,"15min":24.420000000000002},"min":{"1min":9.8610000000000007,"5min":9.1549999999999994,"15min":6.4119999999999999},"max":{"1min":61.780999999999999,"5min":110.056,"15min":279.63999999999999},"last":13.470000000000001}]},{"osd":4,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":13.672000000000001,"5min":18.148,"15min":26.068999999999999},"min":{"1min":7.1959999999999997,"5min":6.2629999999999999,"15min":5.8319999999999999},"max":{"1min":53.085999999999999,"5min":143.24000000000001,"15min":438.75099999999998},"last":7.2919999999999998},{"interface":"front","average":{"1min":18.068999999999999,"5min":21.373000000000001,"15min":31.120000000000001},"min":{"1min":10.132999999999999,"5min":8.4510000000000005,"15min":8.4339999999999993},"max":{"1min":41.640000000000001,"5min":78.664000000000001,"15min":443.58300000000003},"last":12.746}]},{"osd":5,"last update":"Fri Dec 15 17:24:17 2023","interfaces":[{"interface":"back","average":{"1min":27.654,"5min":22.206,"15min":30.274999999999999},"min":{"1min":8.9260000000000002,"5min":7.46,"15min":6.8369999999999997},"max":{"1min":145.34399999999999,"5min":145.34399999999999,"15min":275.17700000000002},"last":8.2699999999999996},{"interface":"front","average":{"1min":34.073999999999998,"5min":29.946999999999999,"15min":27.126000000000001},"min":{"1min":14.428000000000001,"5min":10.959,"15min":6.6500000000000004},"max":{"1min":176.36000000000001,"5min":209.96600000000001,"15min":264.46899999999999},"last":14.853}]},{"osd":6,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":21.939,"5min":22.827999999999999,"15min":31.187999999999999},"min":{"1min":12.606,"5min":8.6809999999999992,"15min":7.5629999999999997},"max":{"1min":90.926000000000002,"5min":162.06200000000001,"15min":455.43000000000001},"last":15.183},{"interface":"front","average":{"1min":19.638999999999999,"5min":21.096,"15min":28.334},"min":{"1min":9.1400000000000006,"5min":7.3399999999999999,"15min":6.7249999999999996},"max":{"1min":94.715999999999994,"5min":169.34,"15min":314.589},"last":9.2669999999999995}]},{"osd":7,"last update":"Fri Dec 15 17:24:17 2023","interfaces":[{"interface":"back","average":{"1min":21.826000000000001,"5min":25.213999999999999,"15min":31.879999999999999},"min":{"1min":13.837,"5min":10.691000000000001,"15min":10.585000000000001},"max":{"1min":36.244,"5min":132.423,"15min":316.85300000000001},"last":16.983000000000001},{"interface":"front","average":{"1min":21.334,"5min":21.224,"15min":25.539999999999999},"min":{"1min":11.933,"5min":10.175000000000001,"15min":8.6110000000000007},"max":{"1min":45.905000000000001,"5min":63.219999999999999,"15min":237.749},"last":17.986000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509714,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58916,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312924,"statfs":{"total":96636764160,"available":96576434176,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":13.765000000000001,"5min":13.314,"15min":18.777000000000001},"min":{"1min":6.2359999999999998,"5min":4.3639999999999999,"15min":4.1429999999999998},"max":{"1min":80.552999999999997,"5min":80.552999999999997,"15min":375.77800000000002},"last":8.0060000000000002},{"interface":"front","average":{"1min":17.832999999999998,"5min":17.524999999999999,"15min":23.969000000000001},"min":{"1min":6.0199999999999996,"5min":4.8170000000000002,"15min":4.8170000000000002},"max":{"1min":94.995999999999995,"5min":94.995999999999995,"15min":556.42700000000002},"last":8.6379999999999999}]},{"osd":2,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":20.041,"5min":19.704999999999998,"15min":21.385999999999999},"min":{"1min":9.3889999999999993,"5min":6.2939999999999996,"15min":6.2939999999999996},"max":{"1min":99.216999999999999,"5min":248.864,"15min":396.13400000000001},"last":12.010999999999999},{"interface":"front","average":{"1min":16.388999999999999,"5min":21.504999999999999,"15min":25.138999999999999},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":5.9790000000000001},"max":{"1min":75.944000000000003,"5min":234.161,"15min":354.649},"last":13.244999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":19.158000000000001,"5min":21.491,"15min":22.245000000000001},"min":{"1min":7.6159999999999997,"5min":6.1740000000000004,"15min":6.0830000000000002},"max":{"1min":96.188000000000002,"5min":99.453000000000003,"15min":413.00599999999997},"last":10.316000000000001},{"interface":"front","average":{"1min":15.550000000000001,"5min":19.518999999999998,"15min":24.094999999999999},"min":{"1min":6.0640000000000001,"5min":6.04,"15min":6.04},"max":{"1min":101.57299999999999,"5min":106.357,"15min":526.98599999999999},"last":6.7359999999999998}]},{"osd":4,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":21.193999999999999,"5min":24.096,"15min":24.780999999999999},"min":{"1min":7.1829999999999998,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":119.754,"5min":266.279,"15min":389.75700000000001},"last":9.2940000000000005},{"interface":"front","average":{"1min":17.59,"5min":22.911999999999999,"15min":25.393000000000001},"min":{"1min":8.4700000000000006,"5min":8.4700000000000006,"15min":7.3520000000000003},"max":{"1min":37.862000000000002,"5min":227.19200000000001,"15min":436.572},"last":10.903}]},{"osd":5,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":17.655000000000001,"5min":22.420000000000002,"15min":26.187999999999999},"min":{"1min":7.8220000000000001,"5min":7.5359999999999996,"15min":7.3540000000000001},"max":{"1min":67.956999999999994,"5min":121.137,"15min":449.81299999999999},"last":13.827999999999999},{"interface":"front","average":{"1min":20.068000000000001,"5min":23.791,"15min":27.331},"min":{"1min":10.81,"5min":7.7439999999999998,"15min":6.2539999999999996},"max":{"1min":65.381,"5min":109.90000000000001,"15min":429.75299999999999},"last":16.632999999999999}]},{"osd":6,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":19.919,"5min":23.132999999999999,"15min":25.440999999999999},"min":{"1min":10.682,"5min":8.6760000000000002,"15min":7.4699999999999998},"max":{"1min":91.619,"5min":93.299000000000007,"15min":469.86799999999999},"last":15.292999999999999},{"interface":"front","average":{"1min":23.565000000000001,"5min":24.164000000000001,"15min":27.797999999999998},"min":{"1min":12.039,"5min":9.2899999999999991,"15min":8.4740000000000002},"max":{"1min":102.664,"5min":102.664,"15min":373.93599999999998},"last":15.875999999999999}]},{"osd":7,"last update":"Fri Dec 15 17:23:41 2023","interfaces":[{"interface":"back","average":{"1min":17.504999999999999,"5min":21.597999999999999,"15min":29.437999999999999},"min":{"1min":8.3109999999999999,"5min":7.0449999999999999,"15min":7.0449999999999999},"max":{"1min":77.521000000000001,"5min":151.999,"15min":416.24400000000003},"last":12.582000000000001},{"interface":"front","average":{"1min":24.347000000000001,"5min":26.122,"15min":27.091000000000001},"min":{"1min":11.058,"5min":9.8610000000000007,"15min":7.7160000000000002},"max":{"1min":100.611,"5min":163.09700000000001,"15min":410.04300000000001},"last":17.739000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509712,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76964,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294876,"statfs":{"total":96636764160,"available":96557953024,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":8.1609999999999996,"5min":13.029,"15min":18.565999999999999},"min":{"1min":5.3899999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"max":{"1min":14.663,"5min":132.791,"15min":252.483},"last":7.6909999999999998},{"interface":"front","average":{"1min":8.7210000000000001,"5min":13.803000000000001,"15min":18.956},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":24.484999999999999,"5min":138.03800000000001,"15min":235.10300000000001},"last":5.5209999999999999}]},{"osd":1,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":10.631,"5min":13.651,"15min":22},"min":{"1min":7.4589999999999996,"5min":4.9390000000000001,"15min":4.9390000000000001},"max":{"1min":19.66,"5min":79.147000000000006,"15min":624.94100000000003},"last":14.824},{"interface":"front","average":{"1min":14.163,"5min":17.541,"15min":24.814},"min":{"1min":8.8680000000000003,"5min":7.4770000000000003,"15min":6.9740000000000002},"max":{"1min":22.484000000000002,"5min":86.152000000000001,"15min":635.13199999999995},"last":15.672000000000001}]},{"osd":2,"last update":"Fri Dec 15 17:23:45 2023","interfaces":[{"interface":"back","average":{"1min":14.015000000000001,"5min":17.581,"15min":22.074999999999999},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":60.210000000000001,"5min":122.60899999999999,"15min":286.22699999999998},"last":8.6940000000000008},{"interface":"front","average":{"1min":13.85,"5min":17.390000000000001,"15min":25.114000000000001},"min":{"1min":6.931,"5min":6.931,"15min":6.5380000000000003},"max":{"1min":73.950000000000003,"5min":238.096,"15min":573.05700000000002},"last":10.555999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:24:04 2023","interfaces":[{"interface":"back","average":{"1min":18.823,"5min":20.178999999999998,"15min":26.625},"min":{"1min":8.5660000000000007,"5min":7.5019999999999998,"15min":7.1159999999999997},"max":{"1min":99.241,"5min":228.708,"15min":628.97299999999996},"last":16.454000000000001},{"interface":"front","average":{"1min":22.285,"5min":24.675999999999998,"15min":31.797999999999998},"min":{"1min":12.448,"5min":11.423,"15min":9.9060000000000006},"max":{"1min":97.656000000000006,"5min":230.202,"15min":637.40200000000004},"last":17.294}]},{"osd":5,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":14.532,"5min":20.297999999999998,"15min":26.827999999999999},"min":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.782},"max":{"1min":32.411000000000001,"5min":144.59200000000001,"15min":752.79499999999996},"last":11.388},{"interface":"front","average":{"1min":16.890999999999998,"5min":21.039000000000001,"15min":25.788},"min":{"1min":9.5,"5min":9.2129999999999992,"15min":6.0119999999999996},"max":{"1min":45.607999999999997,"5min":145.405,"15min":263.79500000000002},"last":9.6370000000000005}]},{"osd":6,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":16.896000000000001,"5min":21.753,"15min":27.977},"min":{"1min":9.3599999999999994,"5min":8.0150000000000006,"15min":7.5819999999999999},"max":{"1min":40.414999999999999,"5min":134.255,"15min":498.464},"last":12.068},{"interface":"front","average":{"1min":21.16,"5min":24.161000000000001,"15min":31.126999999999999},"min":{"1min":12.519,"5min":10.362,"15min":8.6560000000000006},"max":{"1min":60.975000000000001,"5min":153.113,"15min":636.16399999999999},"last":18.045000000000002}]},{"osd":7,"last update":"Fri Dec 15 17:23:26 2023","interfaces":[{"interface":"back","average":{"1min":17.704999999999998,"5min":23.234000000000002,"15min":31.463000000000001},"min":{"1min":11.615,"5min":10.316000000000001,"15min":8.4930000000000003},"max":{"1min":25.109000000000002,"5min":202.89500000000001,"15min":572.08699999999999},"last":13.571999999999999},{"interface":"front","average":{"1min":19.895,"5min":24.616,"15min":31.809999999999999},"min":{"1min":13.664999999999999,"5min":13.173999999999999,"15min":10.406000000000001},"max":{"1min":32.581000000000003,"5min":156.91900000000001,"15min":624.04100000000005},"last":18.798999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542417,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76964,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294876,"statfs":{"total":96636764160,"available":96557953024,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":11.852,"5min":12.436999999999999,"15min":16.131},"min":{"1min":5.0750000000000002,"5min":4.3810000000000002,"15min":4.3259999999999996},"max":{"1min":37.542000000000002,"5min":91.055999999999997,"15min":161.38499999999999},"last":4.891},{"interface":"front","average":{"1min":12.800000000000001,"5min":12.664999999999999,"15min":16.463999999999999},"min":{"1min":6.5910000000000002,"5min":4.8339999999999996,"15min":4.8339999999999996},"max":{"1min":64.691999999999993,"5min":93.209999999999994,"15min":142.673},"last":10.505000000000001}]},{"osd":1,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":20.167000000000002,"5min":18.283000000000001,"15min":22.882000000000001},"min":{"1min":7.2480000000000002,"5min":6.3550000000000004,"15min":5.7629999999999999},"max":{"1min":74.025999999999996,"5min":143.17599999999999,"15min":275.99299999999999},"last":13.394},{"interface":"front","average":{"1min":15.464,"5min":14.946,"15min":21.763999999999999},"min":{"1min":5.9749999999999996,"5min":5.7960000000000003,"15min":5.2649999999999997},"max":{"1min":81.099999999999994,"5min":88.022999999999996,"15min":342.22000000000003},"last":11.505000000000001}]},{"osd":2,"last update":"Fri Dec 15 17:23:37 2023","interfaces":[{"interface":"back","average":{"1min":20.966999999999999,"5min":17.347000000000001,"15min":19.687000000000001},"min":{"1min":8.2509999999999994,"5min":7.383,"15min":6.0670000000000002},"max":{"1min":86.772000000000006,"5min":95.926000000000002,"15min":131.81299999999999},"last":7.524},{"interface":"front","average":{"1min":16.437000000000001,"5min":14.654999999999999,"15min":19.41},"min":{"1min":6.9470000000000001,"5min":6.266,"15min":6.266},"max":{"1min":84.876000000000005,"5min":84.876000000000005,"15min":117.864},"last":12.09}]},{"osd":3,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":20.248000000000001,"5min":20.637,"15min":24.939},"min":{"1min":6.5330000000000004,"5min":6.5330000000000004,"15min":6.4370000000000003},"max":{"1min":118.39400000000001,"5min":118.39400000000001,"15min":390.91000000000003},"last":14.013},{"interface":"front","average":{"1min":20.558,"5min":21.027999999999999,"15min":26.327999999999999},"min":{"1min":8.1479999999999997,"5min":7.4260000000000002,"15min":5.7320000000000002},"max":{"1min":138.559,"5min":138.559,"15min":244.13},"last":8.6910000000000007}]},{"osd":4,"last update":"Fri Dec 15 17:23:51 2023","interfaces":[{"interface":"back","average":{"1min":24.532,"5min":22.248999999999999,"15min":24.129999999999999},"min":{"1min":7.3529999999999998,"5min":7.3529999999999998,"15min":7.3529999999999998},"max":{"1min":144.52099999999999,"5min":144.52099999999999,"15min":245.65600000000001},"last":12.763},{"interface":"front","average":{"1min":25.827999999999999,"5min":24.245999999999999,"15min":23.055},"min":{"1min":9.4199999999999999,"5min":7.726,"15min":6.9139999999999997},"max":{"1min":143.92099999999999,"5min":189.988,"15min":193.07400000000001},"last":17.698}]},{"osd":5,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":17.966000000000001,"5min":19.504000000000001,"15min":25.530000000000001},"min":{"1min":7.7699999999999996,"5min":7.7699999999999996,"15min":7.7699999999999996},"max":{"1min":69.656999999999996,"5min":109.459,"15min":198.96000000000001},"last":17.047000000000001},{"interface":"front","average":{"1min":21.565999999999999,"5min":20.701000000000001,"15min":27.978999999999999},"min":{"1min":8.7959999999999994,"5min":7.6189999999999998,"15min":7.6189999999999998},"max":{"1min":145.94,"5min":165.97999999999999,"15min":266.33699999999999},"last":14.632}]},{"osd":6,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":24.082000000000001,"5min":22.300000000000001,"15min":30.507000000000001},"min":{"1min":11.667999999999999,"5min":7.952,"15min":7.8949999999999996},"max":{"1min":112.369,"5min":113.128,"15min":380.62700000000001},"last":18.844000000000001},{"interface":"front","average":{"1min":23.890999999999998,"5min":21.359000000000002,"15min":26.186},"min":{"1min":8.3550000000000004,"5min":8.3550000000000004,"15min":7.2270000000000003},"max":{"1min":108.03,"5min":108.03,"15min":226.91499999999999},"last":15.327}]}]},{"osd":6,"up_from":14,"seq":60129542418,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76964,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294876,"statfs":{"total":96636764160,"available":96557953024,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":10.004,"5min":12.342000000000001,"15min":18.960000000000001},"min":{"1min":4.4269999999999996,"5min":4.133,"15min":4.133},"max":{"1min":25.085999999999999,"5min":128.82499999999999,"15min":223.15199999999999},"last":7.3330000000000002},{"interface":"front","average":{"1min":9.6329999999999991,"5min":14.333,"15min":18.896000000000001},"min":{"1min":6.2009999999999996,"5min":5.3490000000000002,"15min":4.6920000000000002},"max":{"1min":17.561,"5min":120.85299999999999,"15min":190.249},"last":12.015000000000001}]},{"osd":1,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":12.476000000000001,"5min":18.774000000000001,"15min":22.052},"min":{"1min":7.2279999999999998,"5min":5.1319999999999997,"15min":5.0449999999999999},"max":{"1min":24.771999999999998,"5min":177.15100000000001,"15min":292.72199999999998},"last":9.5129999999999999},{"interface":"front","average":{"1min":14.473000000000001,"5min":19.847999999999999,"15min":25.225000000000001},"min":{"1min":7.6120000000000001,"5min":6.4390000000000001,"15min":5.0309999999999997},"max":{"1min":41.270000000000003,"5min":174.02199999999999,"15min":422.36099999999999},"last":10.795999999999999}]},{"osd":2,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":17.154,"5min":17.030000000000001,"15min":19.913},"min":{"1min":7.7169999999999996,"5min":6.2850000000000001,"15min":5.2699999999999996},"max":{"1min":66.974000000000004,"5min":131.71000000000001,"15min":131.863},"last":14.132},{"interface":"front","average":{"1min":13.039,"5min":19.218,"15min":23.687000000000001},"min":{"1min":6.9530000000000003,"5min":6.9530000000000003,"15min":6.7439999999999998},"max":{"1min":65.632000000000005,"5min":112.548,"15min":189.417},"last":7.9850000000000003}]},{"osd":3,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":21.212,"5min":24.562000000000001,"15min":27.657},"min":{"1min":7.7279999999999998,"5min":7.7279999999999998,"15min":7.7279999999999998},"max":{"1min":77.271000000000001,"5min":263.45299999999997,"15min":263.45299999999997},"last":17.021999999999998},{"interface":"front","average":{"1min":24.375,"5min":27.704000000000001,"15min":30.245999999999999},"min":{"1min":11.551,"5min":10.195,"15min":6.9100000000000001},"max":{"1min":83.891999999999996,"5min":264.54000000000002,"15min":356.815},"last":17.995999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:24:28 2023","interfaces":[{"interface":"back","average":{"1min":27.597999999999999,"5min":21.550000000000001,"15min":22.762},"min":{"1min":10.119999999999999,"5min":6.4020000000000001,"15min":6.4020000000000001},"max":{"1min":127.682,"5min":127.682,"15min":183.25299999999999},"last":10.119999999999999},{"interface":"front","average":{"1min":16.852,"5min":18.989999999999998,"15min":24.437000000000001},"min":{"1min":8.9019999999999992,"5min":6.9100000000000001,"15min":6.9100000000000001},"max":{"1min":59.033000000000001,"5min":106.631,"15min":266.88400000000001},"last":12.685}]},{"osd":5,"last update":"Fri Dec 15 17:23:38 2023","interfaces":[{"interface":"back","average":{"1min":15.765000000000001,"5min":21.884,"15min":28.344000000000001},"min":{"1min":7.9089999999999998,"5min":7.601,"15min":7.0369999999999999},"max":{"1min":40.454999999999998,"5min":164.62899999999999,"15min":349.983},"last":13.452999999999999},{"interface":"front","average":{"1min":14.981999999999999,"5min":21.09,"15min":25.739000000000001},"min":{"1min":9.1750000000000007,"5min":7.4530000000000003,"15min":6.8449999999999998},"max":{"1min":26.895,"5min":115.54900000000001,"15min":324.95299999999997},"last":14.871}]},{"osd":7,"last update":"Fri Dec 15 17:23:47 2023","interfaces":[{"interface":"back","average":{"1min":15.082000000000001,"5min":22.818999999999999,"15min":28.507000000000001},"min":{"1min":9.4640000000000004,"5min":8.2110000000000003,"15min":6.8520000000000003},"max":{"1min":38.795999999999999,"5min":137.59700000000001,"15min":268.66500000000002},"last":16.402000000000001},{"interface":"front","average":{"1min":18.492999999999999,"5min":26.584,"15min":33.052999999999997},"min":{"1min":12.459,"5min":8.8849999999999998,"15min":8.8849999999999998},"max":{"1min":48.68,"5min":135.74199999999999,"15min":318.56900000000002},"last":18.757000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542419,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81516,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290324,"statfs":{"total":96636764160,"available":96553291776,"internally_reserved":0,"allocated":6066176,"data_stored":3403689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":10.813000000000001,"5min":18.579999999999998,"15min":19.327999999999999},"min":{"1min":5.0339999999999998,"5min":4.7380000000000004,"15min":4.2859999999999996},"max":{"1min":24.373000000000001,"5min":209.15799999999999,"15min":223.69499999999999},"last":7.9429999999999996},{"interface":"front","average":{"1min":10.352,"5min":17.178999999999998,"15min":19.341999999999999},"min":{"1min":4.2590000000000003,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":17.353000000000002,"5min":183.93899999999999,"15min":226.76300000000001},"last":14.246}]},{"osd":1,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":13.563000000000001,"5min":17.940000000000001,"15min":22.196000000000002},"min":{"1min":6.6539999999999999,"5min":5.1260000000000003,"15min":4.7839999999999998},"max":{"1min":63.030999999999999,"5min":137.07499999999999,"15min":340.32999999999998},"last":17.123999999999999},{"interface":"front","average":{"1min":18.015999999999998,"5min":20.047999999999998,"15min":24.577000000000002},"min":{"1min":8.7149999999999999,"5min":5.7850000000000001,"15min":5.6550000000000002},"max":{"1min":57.795000000000002,"5min":169.648,"15min":326.375},"last":8.7230000000000008}]},{"osd":2,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":17.568000000000001,"5min":20.483000000000001,"15min":22.577000000000002},"min":{"1min":6.6630000000000003,"5min":6.6630000000000003,"15min":5.3979999999999997},"max":{"1min":92.230000000000004,"5min":174.09100000000001,"15min":240.00999999999999},"last":9.8040000000000003},{"interface":"front","average":{"1min":21.774999999999999,"5min":24.068000000000001,"15min":25.559000000000001},"min":{"1min":6.7690000000000001,"5min":5.569,"15min":5.569},"max":{"1min":128.98099999999999,"5min":228.31899999999999,"15min":371.30099999999999},"last":12.619999999999999}]},{"osd":3,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":25.259,"5min":25.724,"15min":28.169},"min":{"1min":9.2319999999999993,"5min":6.5999999999999996,"15min":6.5999999999999996},"max":{"1min":100.711,"5min":177.51300000000001,"15min":321.26100000000002},"last":13.505000000000001},{"interface":"front","average":{"1min":25.388999999999999,"5min":26.262,"15min":27.506},"min":{"1min":7.3799999999999999,"5min":7.101,"15min":6.8730000000000002},"max":{"1min":101.599,"5min":172.887,"15min":284.12599999999998},"last":11.481999999999999}]},{"osd":4,"last update":"Fri Dec 15 17:23:40 2023","interfaces":[{"interface":"back","average":{"1min":18.481000000000002,"5min":29.667999999999999,"15min":27.547000000000001},"min":{"1min":8.3559999999999999,"5min":8.3559999999999999,"15min":6.8600000000000003},"max":{"1min":61.365000000000002,"5min":229.077,"15min":376.39800000000002},"last":18.986999999999998},{"interface":"front","average":{"1min":19.219000000000001,"5min":28.552,"15min":27.09},"min":{"1min":10.989000000000001,"5min":7.8840000000000003,"15min":6.2789999999999999},"max":{"1min":55.509999999999998,"5min":211.03100000000001,"15min":338.56400000000002},"last":10.805}]},{"osd":6,"last update":"Fri Dec 15 17:23:29 2023","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":25.266999999999999,"15min":27.039000000000001},"min":{"1min":12.606999999999999,"5min":7.8079999999999998,"15min":7.0789999999999997},"max":{"1min":57.170000000000002,"5min":215.21299999999999,"15min":378.327},"last":14.962},{"interface":"front","average":{"1min":21.053000000000001,"5min":29.216999999999999,"15min":28.443000000000001},"min":{"1min":9.3049999999999997,"5min":8.8900000000000006,"15min":6.6369999999999996},"max":{"1min":62.055999999999997,"5min":228.58799999999999,"15min":407.16500000000002},"last":19.864000000000001}]},{"osd":7,"last update":"Fri Dec 15 17:23:35 2023","interfaces":[{"interface":"back","average":{"1min":19.960999999999999,"5min":24.295000000000002,"15min":28.638999999999999},"min":{"1min":12.951000000000001,"5min":10.864000000000001,"15min":7.7599999999999998},"max":{"1min":39.515999999999998,"5min":151.386,"15min":292.82299999999998},"last":17.856000000000002},{"interface":"front","average":{"1min":20.140000000000001,"5min":27.221,"15min":30.507000000000001},"min":{"1min":11.416,"5min":9.2170000000000005,"15min":7.8650000000000002},"max":{"1min":28.832999999999998,"5min":190.74600000000001,"15min":312.73599999999999},"last":21.119}]}]},{"osd":4,"up_from":14,"seq":60129542419,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58916,"kb_used_data":5532,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312924,"statfs":{"total":96636764160,"available":96576434176,"internally_reserved":0,"allocated":5664768,"data_stored":3005849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":20.422000000000001,"5min":13.071,"15min":15.058},"min":{"1min":4.7779999999999996,"5min":4.4560000000000004,"15min":4.1289999999999996},"max":{"1min":62.142000000000003,"5min":62.142000000000003,"15min":193.11500000000001},"last":11.157999999999999},{"interface":"front","average":{"1min":27.135999999999999,"5min":13.522,"15min":16.253},"min":{"1min":5.9699999999999998,"5min":4.4109999999999996,"15min":4.3719999999999999},"max":{"1min":96.224000000000004,"5min":96.224000000000004,"15min":235.09800000000001},"last":9.8390000000000004}]},{"osd":1,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":26.41,"5min":16.433,"15min":19.759},"min":{"1min":8.8640000000000008,"5min":5.2199999999999998,"15min":4.9930000000000003},"max":{"1min":86.936999999999998,"5min":86.936999999999998,"15min":289.11099999999999},"last":15.503},{"interface":"front","average":{"1min":22.891999999999999,"5min":18.584,"15min":21.152999999999999},"min":{"1min":7.3520000000000003,"5min":6.2549999999999999,"15min":6.2249999999999996},"max":{"1min":83.016999999999996,"5min":130.535,"15min":218.32400000000001},"last":7.1900000000000004}]},{"osd":2,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":35.674999999999997,"5min":20.347999999999999,"15min":18.050000000000001},"min":{"1min":8.2569999999999997,"5min":6.8179999999999996,"15min":5.5880000000000001},"max":{"1min":163.31399999999999,"5min":163.31399999999999,"15min":163.31399999999999},"last":8.7420000000000009},{"interface":"front","average":{"1min":31.010000000000002,"5min":19.277999999999999,"15min":18.506},"min":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":5.4690000000000003},"max":{"1min":134.78700000000001,"5min":134.78700000000001,"15min":140.416},"last":14.816000000000001}]},{"osd":3,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":26.495000000000001,"5min":23.353000000000002,"15min":21.949000000000002},"min":{"1min":11.071999999999999,"5min":8.9169999999999998,"15min":7.2450000000000001},"max":{"1min":89.188000000000002,"5min":230.505,"15min":230.505},"last":13.436},{"interface":"front","average":{"1min":22.827000000000002,"5min":19.329999999999998,"15min":21.117999999999999},"min":{"1min":9.0299999999999994,"5min":6.4210000000000003,"15min":6.4210000000000003},"max":{"1min":84.497,"5min":136.66499999999999,"15min":203.99100000000001},"last":16.861000000000001}]},{"osd":5,"last update":"Fri Dec 15 17:23:30 2023","interfaces":[{"interface":"back","average":{"1min":19.713000000000001,"5min":19.503,"15min":21.353999999999999},"min":{"1min":7.7530000000000001,"5min":6.9930000000000003,"15min":6.9930000000000003},"max":{"1min":87.989999999999995,"5min":123.71599999999999,"15min":162.02099999999999},"last":14.129},{"interface":"front","average":{"1min":23.210999999999999,"5min":23.539999999999999,"15min":24.937999999999999},"min":{"1min":10.042999999999999,"5min":10.042999999999999,"15min":7.1130000000000004},"max":{"1min":89.099999999999994,"5min":159.16999999999999,"15min":162.922},"last":20.390999999999998}]},{"osd":6,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":25.030000000000001,"5min":19.335999999999999,"15min":22.289999999999999},"min":{"1min":7.5579999999999998,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":84.040000000000006,"5min":85.606999999999999,"15min":744.33199999999999},"last":11.842000000000001},{"interface":"front","average":{"1min":38.479999999999997,"5min":26.033000000000001,"15min":25.606000000000002},"min":{"1min":14.532999999999999,"5min":10.544,"15min":9.1799999999999997},"max":{"1min":157.441,"5min":157.441,"15min":251.19200000000001},"last":16.187999999999999}]},{"osd":7,"last update":"Fri Dec 15 17:23:43 2023","interfaces":[{"interface":"back","average":{"1min":31.425999999999998,"5min":25.463000000000001,"15min":26.760999999999999},"min":{"1min":17.059000000000001,"5min":11.148999999999999,"15min":9.7330000000000005},"max":{"1min":91.006,"5min":144.72,"15min":202.46600000000001},"last":19.559000000000001},{"interface":"front","average":{"1min":21.945,"5min":18.006,"15min":20.908000000000001},"min":{"1min":9.6690000000000005,"5min":7.5860000000000003,"15min":6.5430000000000001},"max":{"1min":78.950000000000003,"5min":85.629999999999995,"15min":199.01400000000001},"last":18.416}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-15T17:24:30.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-15T17:24:30.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.340+0000 7f63ca2a2640 10 monclient: get_monmap_and_config 2023-12-15T17:24:30.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.340+0000 7f63ca2a2640 10 monclient: build_initial_monmap 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.340+0000 7f63ca2a2640 10 monclient: monmap: 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:24:30.341685+0000 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:24:30.341685+0000 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c-legacy 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.192:3300/0 mon.noname-c 2023-12-15T17:24:30.342 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.340+0000 7f63ca2a2640 1 Processor -- start 2023-12-15T17:24:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.340+0000 7f63ca2a2640 1 -- start start 2023-12-15T17:24:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.340+0000 7f63ca2a2640 10 monclient: init 2023-12-15T17:24:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-12-15T17:24:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 0x7f63c406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 10 monclient(hunting): picked mon.noname-b con 0x7f63c406cec0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63c4071730 addr v1:172.21.15.69:6789/0 2023-12-15T17:24:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63c413eff0 addr v1:172.21.15.192:6789/0 2023-12-15T17:24:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c4070dc0 con 0x7f63c4071730 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c4071030 con 0x7f63c413eff0 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4071310 con 0x7f63c406cec0 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.341+0000 7f63ca2a2640 10 monclient(hunting): authenticate will time out at 2774.657955s 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.342+0000 7f63c3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 0x7f63c406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.342+0000 7f63c37fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f63c4071730 0x7f63c4071b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43362/0 (socket says 172.21.15.69:43362) 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.342+0000 7f63c37fe640 1 -- 172.21.15.69:0/474873238 learned_addr learned my addr 172.21.15.69:0/474873238 (peer_addr_for_me v1:172.21.15.69:0/0) 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.342+0000 7f63c3fff640 10 monclient(hunting): get_auth_request con 0x7f63c406cec0 auth_method 0 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.342+0000 7f63c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.342+0000 7f63c3fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.342+0000 7f63c3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.342+0000 7f63c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.342+0000 7f63c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.342+0000 7f63c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:30.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.344+0000 7f63c3fff640 10 monclient(hunting): handle_auth_done global_id 17867 payload 274 2023-12-15T17:24:30.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.344+0000 7f63c3fff640 1 -- 172.21.15.69:0/474873238 >> v1:172.21.15.192:6789/0 conn(0x7f63c413eff0 legacy=0x7f63c413f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:30.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.344+0000 7f63c3fff640 1 -- 172.21.15.69:0/474873238 >> v1:172.21.15.69:6789/0 conn(0x7f63c4071730 legacy=0x7f63c4071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:30.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.344+0000 7f63c3fff640 10 monclient: _finish_hunting 0 2023-12-15T17:24:30.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.344+0000 7f63c3fff640 1 monclient: found mon.noname-b 2023-12-15T17:24:30.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.344+0000 7f63c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:30.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.344+0000 7f63c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2023-12-15T17:24:30.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.344+0000 7f63c3fff640 1 -- 172.21.15.69:0/474873238 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c406c2a0 con 0x7f63c406cec0 2023-12-15T17:24:30.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.346+0000 7f63c3fff640 1 --2- 172.21.15.69:0/474873238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 0x7f63c406d2a0 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f63b80097d0 tx=0x7f63b802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52559d729c5275a6 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:30.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 1 -- 172.21.15.69:0/474873238 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8031890 con 0x7f63c406cec0 2023-12-15T17:24:30.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:30.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-15T17:24:30.351 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 10 monclient: dump: 2023-12-15T17:24:30.351 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:30.351 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 10 monclient: _finish_auth 0 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 10 monclient: _check_auth_tickets 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 1 -- 172.21.15.69:0/474873238 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b80319f0 con 0x7f63c406cec0 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 1 -- 172.21.15.69:0/474873238 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8031d80 con 0x7f63c406cec0 2023-12-15T17:24:30.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63ca2a2640 5 monclient: authenticate success, global_id 17867 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63ca2a2640 10 monclient: get_monmap_and_config success 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63ca2a2640 4 set_mon_vals no callback set 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.349+0000 7f63c2ffd640 10 monclient: dump: 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:30.353 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:30.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.350+0000 7f63ca2a2640 10 monclient: shutdown 2023-12-15T17:24:30.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.350+0000 7f63ca2a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:30.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.350+0000 7f63ca2a2640 1 -- 172.21.15.69:0/474873238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 msgr2=0x7f63c406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:30.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.350+0000 7f63ca2a2640 1 --2- 172.21.15.69:0/474873238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 0x7f63c406d2a0 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f63b80097d0 tx=0x7f63b802b930 comp rx=0 tx=0).stop 2023-12-15T17:24:30.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.350+0000 7f63ca2a2640 1 -- 172.21.15.69:0/474873238 shutdown_connections 2023-12-15T17:24:30.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.350+0000 7f63ca2a2640 1 --2- 172.21.15.69:0/474873238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 0x7f63c406d2a0 unknown :-1 s=CLOSED pgs=3139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:30.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.350+0000 7f63ca2a2640 1 -- 172.21.15.69:0/474873238 >> 172.21.15.69:0/474873238 conn(0x7f63c40b5ca0 msgr2=0x7f63c40b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.350+0000 7f63ca2a2640 1 -- 172.21.15.69:0/474873238 shutdown_connections 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.350+0000 7f63ca2a2640 1 -- 172.21.15.69:0/474873238 wait complete. 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.351+0000 7f63ca2a2640 10 monclient: build_initial_monmap 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.351+0000 7f63ca2a2640 10 monclient: monmap: 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.noname-b 2023-12-15T17:24:30.358 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 librados: starting msgr at 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 librados: starting objecter 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 Processor -- start 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 -- start start 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 librados: setting wanted keys 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 librados: calling monclient init 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 10 monclient: init 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 10 monclient: _reopen_session rank -1 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 0x7f63c4138840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 10 monclient(hunting): picked mon.noname-c con 0x7f63c406cec0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63c4071730 0x7f63c414cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 10 monclient(hunting): picked mon.noname-b con 0x7f63c4071730 addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c413eff0 0x7f63c413b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:30.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 10 monclient(hunting): picked mon.noname-a con 0x7f63c413eff0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40af7e0 con 0x7f63c413eff0 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4138dd0 con 0x7f63c4071730 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 10 monclient(hunting): start opening mon connection 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40aea60 con 0x7f63c406cec0 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 10 monclient(hunting): _renew_subs 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.352+0000 7f63ca2a2640 10 monclient(hunting): authenticate will time out at 2774.669245s 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 0x7f63c4138840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c883f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c413eff0 0x7f63c413b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c37fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63c4071730 0x7f63c414cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 0x7f63c4138840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46240/0 (socket says 172.21.15.69:46240) 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c3fff640 1 -- 172.21.15.69:0/906912821 learned_addr learned my addr 172.21.15.69:0/906912821 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c3fff640 10 monclient(hunting): get_auth_request con 0x7f63c406cec0 auth_method 0 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c3fff640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c883f640 10 monclient(hunting): get_auth_request con 0x7f63c413eff0 auth_method 0 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c883f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c883f640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c883f640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c37fe640 10 monclient(hunting): get_auth_request con 0x7f63c4071730 auth_method 0 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c37fe640 10 monclient(hunting): _init_auth method 2 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.353+0000 7f63c37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.354+0000 7f63c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.354+0000 7f63c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.354+0000 7f63c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.354+0000 7f63c883f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.354+0000 7f63c883f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.354+0000 7f63c883f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.354+0000 7f63c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-15T17:24:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.354+0000 7f63c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-15T17:24:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.354+0000 7f63c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-15T17:24:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.356+0000 7f63c3fff640 10 monclient(hunting): handle_auth_done global_id 17870 payload 824 2023-12-15T17:24:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.356+0000 7f63c3fff640 1 -- 172.21.15.69:0/906912821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63c4071730 msgr2=0x7f63c414cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.356+0000 7f63c3fff640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63c4071730 0x7f63c414cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.356+0000 7f63c3fff640 1 -- 172.21.15.69:0/906912821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c413eff0 msgr2=0x7f63c413b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.356+0000 7f63c3fff640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c413eff0 0x7f63c413b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.356+0000 7f63c3fff640 10 monclient: _finish_hunting 0 2023-12-15T17:24:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.356+0000 7f63c3fff640 1 monclient: found mon.noname-c 2023-12-15T17:24:30.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.356+0000 7f63c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-15T17:24:30.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.356+0000 7f63c3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2023-12-15T17:24:30.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.356+0000 7f63c3fff640 1 -- 172.21.15.69:0/906912821 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b8009400 con 0x7f63c406cec0 2023-12-15T17:24:30.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.359+0000 7f63c3fff640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 0x7f63c4138840 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7f63b80098f0 tx=0x7f63b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:30.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.359+0000 7f63c0ff9640 10 client.?.objecter ms_handle_connect 0x7f63c406cec0 2023-12-15T17:24:30.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.359+0000 7f63c0ff9640 10 client.?.objecter resend_mon_ops 2023-12-15T17:24:30.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 1 -- 172.21.15.69:0/906912821 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80453f0 con 0x7f63c406cec0 2023-12-15T17:24:30.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:30.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-15T17:24:30.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 10 monclient: dump: 2023-12-15T17:24:30.364 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:30.364 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:30.364 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 10 monclient: _finish_auth 0 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 10 monclient: _check_auth_tickets 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 1 -- 172.21.15.69:0/906912821 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8045590 con 0x7f63c406cec0 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63ca2a2640 5 monclient: authenticate success, global_id 17870 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63ca2a2640 10 monclient: _renew_subs 2023-12-15T17:24:30.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63ca2a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63ca2a2640 1 -- 172.21.15.69:0/906912821 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c413b790 con 0x7f63c406cec0 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 1 -- 172.21.15.69:0/906912821 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8045940 con 0x7f63c406cec0 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c0ff9640 10 monclient: dump: 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 1a9f4b8c-183d-4435-93ef-5057f2b4aded 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:created 2023-12-15T17:00:17.127228+0000 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-12-15T17:24:30.366 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-12-15T17:24:30.367 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:24:30.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63ca2a2640 10 client.17870.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-15T17:24:30.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63ca2a2640 10 monclient: _renew_subs 2023-12-15T17:24:30.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63c27fc640 4 set_mon_vals no callback set 2023-12-15T17:24:30.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63ca2a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:30.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63ca2a2640 1 -- 172.21.15.69:0/906912821 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c413bbe0 con 0x7f63c406cec0 2023-12-15T17:24:30.367 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.362+0000 7f63ca2a2640 1 librados: init done 2023-12-15T17:24:30.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.363+0000 7f63ca2a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:30.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.363+0000 7f63ca2a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:30.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.363+0000 7f63ca2a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:30.368 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.363+0000 7f63ca2a2640 1 -- 172.21.15.69:0/906912821 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c413ab00 con 0x7f63c406cec0 2023-12-15T17:24:30.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.382+0000 7f63c0ff9640 1 -- 172.21.15.69:0/906912821 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f63b804e450 con 0x7f63c406cec0 2023-12-15T17:24:30.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.383+0000 7f63c0ff9640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f63b006cba0 0x7f63b006f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-15T17:24:30.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.383+0000 7f63c37fe640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f63b006cba0 0x7f63b006f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-15T17:24:30.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.383+0000 7f63c37fe640 10 monclient: get_auth_request con 0x7f63b006cba0 auth_method 0 2023-12-15T17:24:30.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.384+0000 7f63c0ff9640 10 client.17870.objecter ms_handle_connect 0x7f63b006cba0 2023-12-15T17:24:30.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.384+0000 7f63c37fe640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f63b006cba0 0x7f63b006f060 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f63c4072c80 tx=0x7f63b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-15T17:24:30.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.620+0000 7f63c0ff9640 1 -- 172.21.15.69:0/906912821 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(908..908 src has 718..908) v4 ==== 6185+0+0 (secure 0 0 0) 0x7f63b807ad40 con 0x7f63c406cec0 2023-12-15T17:24:30.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.620+0000 7f63c0ff9640 10 client.17870.objecter ms_dispatch 0x7f63c4000b90 osd_map(908..908 src has 718..908) v4 2023-12-15T17:24:30.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.620+0000 7f63c0ff9640 3 client.17870.objecter handle_osd_map got epochs [908,908] > 0 2023-12-15T17:24:30.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.620+0000 7f63c0ff9640 3 client.17870.objecter handle_osd_map decoding full epoch 908 2023-12-15T17:24:30.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.620+0000 7f63c0ff9640 20 client.17870.objecter dump_active .. 0 homeless 2023-12-15T17:24:30.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.620+0000 7f63c0ff9640 1 -- 172.21.15.69:0/906912821 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f63b8077080 con 0x7f63c406cec0 2023-12-15T17:24:30.622 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.620+0000 7f63c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-15T17:24:30.623 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.620+0000 7f63c0ff9640 10 monclient: _finish_command 1 = system:0 2023-12-15T17:24:30.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.790+0000 7f63ca2a2640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-15T17:24:30.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.790+0000 7f63ca2a2640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-15T17:24:30.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.790+0000 7f63ca2a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2023-12-15T17:24:30.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:30.790+0000 7f63ca2a2640 1 -- 172.21.15.69:0/906912821 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f63c413ab00 con 0x7f63c406cec0 2023-12-15T17:24:31.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.012+0000 7f63c0ff9640 1 -- 172.21.15.69:0/906912821 <== mon.2 v2:172.21.15.69:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f63b8077080 con 0x7f63c406cec0 2023-12-15T17:24:31.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.012+0000 7f63c0ff9640 10 monclient: handle_config config(1 keys) v1 2023-12-15T17:24:31.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.012+0000 7f63c27fc640 4 set_mon_vals no callback set 2023-12-15T17:24:31.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.012+0000 7f63c27fc640 10 set_mon_vals mon_health_to_clog = false 2023-12-15T17:24:31.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.012+0000 7f63c0ff9640 1 -- 172.21.15.69:0/906912821 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f63c413ab00 con 0x7f63c406cec0 2023-12-15T17:24:31.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.012+0000 7f63c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-15T17:24:31.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.012+0000 7f63c0ff9640 10 monclient: _finish_command 2 = system:0 2023-12-15T17:24:31.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.014+0000 7f63ca2a2640 10 librados: watch_flush enter 2023-12-15T17:24:31.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.014+0000 7f63ca2a2640 10 librados: watch_flush exit 2023-12-15T17:24:31.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.014+0000 7f63ca2a2640 20 client.17870.objecter shutdown clearing up homeless session... 2023-12-15T17:24:31.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.014+0000 7f63ca2a2640 10 client.17870.objecter successfully canceled tick 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.014+0000 7f63ca2a2640 1 -- 172.21.15.69:0/906912821 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f63b006cba0 msgr2=0x7f63b006f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.014+0000 7f63ca2a2640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f63b006cba0 0x7f63b006f060 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f63c4072c80 tx=0x7f63b4009040 comp rx=0 tx=0).stop 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.014+0000 7f63ca2a2640 10 monclient: shutdown 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.014+0000 7f63ca2a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.014+0000 7f63ca2a2640 1 -- 172.21.15.69:0/906912821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 msgr2=0x7f63c4138840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.014+0000 7f63ca2a2640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 0x7f63c4138840 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7f63b80098f0 tx=0x7f63b80046c0 comp rx=0 tx=0).stop 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.014+0000 7f63ca2a2640 1 -- 172.21.15.69:0/906912821 shutdown_connections 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.015+0000 7f63ca2a2640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.192:6800/4171480121,v1:172.21.15.192:6801/4171480121] conn(0x7f63b006cba0 0x7f63b006f060 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.015+0000 7f63ca2a2640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c413eff0 0x7f63c413b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.015+0000 7f63ca2a2640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63c4071730 0x7f63c414cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.015+0000 7f63ca2a2640 1 --2- 172.21.15.69:0/906912821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406cec0 0x7f63c4138840 unknown :-1 s=CLOSED pgs=3140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.015+0000 7f63ca2a2640 1 -- 172.21.15.69:0/906912821 >> 172.21.15.69:0/906912821 conn(0x7f63c40b5ca0 msgr2=0x7f63c40b0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.015+0000 7f63ca2a2640 1 -- 172.21.15.69:0/906912821 shutdown_connections 2023-12-15T17:24:31.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.015+0000 7f63ca2a2640 1 -- 172.21.15.69:0/906912821 wait complete. 2023-12-15T17:24:31.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-15T17:24:31.015+0000 7f63ca2a2640 1 librados: shutdown 2023-12-15T17:24:31.030 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-15T17:24:31.030 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-15T17:24:31.030 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-15T17:24:31.030 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:24:31.042 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:24:31.033+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86731) UID: 0 2023-12-15T17:24:31.044 INFO:tasks.ceph.osd.0.smithi069.stderr:2023-12-15T17:24:31.034+0000 b47e640 -1 osd.0 908 *** Got signal Terminated *** 2023-12-15T17:24:33.789 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:43.735 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:33.793 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:43.739 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:33.815 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:43.761 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:33.834 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:43.774 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:33.838 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:43.778 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:33.846 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:43.791 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:33.847 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:43.793 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:33.855 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:43.795 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:33.868 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:43.814 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:34.479 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:44.419 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:34.480 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:44.420 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:34.488 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:44.428 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:34.615 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:44.560 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:34.615 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:44.560 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:34.616 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:44.561 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:34.616 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:44.561 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:34.624 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:44.570 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:34.629 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:44.575 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:53.865 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:03.810 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:53.867 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:03.813 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:53.885 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:03.830 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:53.914 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:03.860 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:53.916 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:03.855 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:53.918 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:03.858 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:53.920 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:03.866 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:53.935 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:03.874 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:53.937 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:03.882 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:54.559 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:04.498 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:54.560 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:04.499 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:54.569 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:04.508 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:54.604 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:04.550 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:54.605 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:04.551 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:54.613 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:04.559 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:54.643 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:04.589 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:54.644 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:04.590 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:24:54.652 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:04.598 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:01.537 INFO:tasks.ceph.osd.0:Stopped 2023-12-15T17:25:01.537 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-15T17:25:01.537 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:25:01.550 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-12-15T17:25:01.541+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86742) UID: 0 2023-12-15T17:25:01.550 INFO:tasks.ceph.osd.1.smithi069.stderr:2023-12-15T17:25:01.543+0000 b47e640 -1 osd.1 912 *** Got signal Terminated *** 2023-12-15T17:25:02.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-12-15T17:25:03.294 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:13.240 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:03.298 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:13.243 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:03.314 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:13.260 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:03.349 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:13.289 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:03.350 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:13.295 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:03.350 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:13.290 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:03.351 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:13.297 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:03.365 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:13.304 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:03.368 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:13.313 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:07.639 INFO:tasks.ceph.osd.1:Stopped 2023-12-15T17:25:07.640 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-15T17:25:07.640 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:25:07.652 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:25:07.643+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86738) UID: 0 2023-12-15T17:25:07.655 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-12-15T17:25:07.644+0000 b47e640 -1 osd.2 915 *** Got signal Terminated *** 2023-12-15T17:25:08.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-15T17:25:08.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-12-15T17:25:13.741 INFO:tasks.ceph.osd.2:Stopped 2023-12-15T17:25:13.742 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-15T17:25:13.742 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:25:13.754 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-12-15T17:25:13.745+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86744) UID: 0 2023-12-15T17:25:13.754 INFO:tasks.ceph.osd.3.smithi069.stderr:2023-12-15T17:25:13.746+0000 b47e640 -1 osd.3 917 *** Got signal Terminated *** 2023-12-15T17:25:14.112 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:24.058 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:14.119 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:24.064 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:14.139 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:24.085 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:14.168 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:24.113 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:14.171 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:24.111 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:14.172 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:24.112 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:14.173 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:24.119 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:14.188 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:24.128 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:14.191 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:24.136 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:14.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-12-15T17:25:14.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-12-15T17:25:19.844 INFO:tasks.ceph.osd.3:Stopped 2023-12-15T17:25:19.844 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-15T17:25:19.844 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:25:19.860 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:25:19.848+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86798) UID: 0 2023-12-15T17:25:19.860 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-12-15T17:25:19.849+0000 b47e640 -1 osd.4 920 *** Got signal Terminated *** 2023-12-15T17:25:20.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-12-15T17:25:20.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-12-15T17:25:25.946 INFO:tasks.ceph.osd.4:Stopped 2023-12-15T17:25:25.946 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-15T17:25:25.946 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:25:25.958 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-12-15T17:25:25.949+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86799) UID: 0 2023-12-15T17:25:25.960 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-12-15T17:25:25.950+0000 b47e640 -1 osd.5 924 *** Got signal Terminated *** 2023-12-15T17:25:26.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-12-15T17:25:26.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-12-15T17:25:29.024 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:38.969 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:29.069 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:39.015 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:29.091 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:39.036 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:29.113 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:39.053 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:29.116 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:39.062 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:29.117 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:39.056 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:29.120 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:39.066 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:29.136 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:39.076 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:29.145 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:39.091 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:31.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-12-15T17:25:34.095 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:44.040 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:34.099 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:44.044 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:34.111 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:44.056 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:34.138 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:44.078 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:34.144 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:44.083 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:34.147 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:44.093 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:34.149 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:44.095 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:34.153 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:44.093 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:34.160 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:44.106 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:49.116 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:59.061 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:49.121 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:59.066 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:49.136 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:59.082 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:49.167 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:59.113 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:49.168 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:59.108 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:49.171 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:59.117 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:49.171 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:59.111 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:49.184 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:59.130 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:49.186 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:59.125 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:54.087 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:04.033 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:54.141 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:04.087 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:54.154 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:04.099 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:54.183 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:04.122 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:54.186 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:04.125 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:54.191 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:04.136 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:54.196 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:04.142 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:54.196 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:04.136 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:25:54.207 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:04.153 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:06.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-12-15T17:26:08.655 INFO:tasks.ceph.osd.5:Stopped 2023-12-15T17:26:08.656 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-15T17:26:08.656 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:26:08.669 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-12-15T17:26:08.660+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86801) UID: 0 2023-12-15T17:26:08.671 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-12-15T17:26:08.661+0000 b47e640 -1 osd.6 926 *** Got signal Terminated *** 2023-12-15T17:26:09.173 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:19.118 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:09.176 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:19.121 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:09.191 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:19.137 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:09.222 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:19.162 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:09.223 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:19.169 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:09.224 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:19.164 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:09.226 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:19.172 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:09.238 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:19.178 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:09.241 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:19.186 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:12.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-12-15T17:26:14.250 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:24.195 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:14.255 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:24.201 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:14.300 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:24.239 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:14.301 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:24.247 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:14.304 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:24.250 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:14.307 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:24.253 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:14.308 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:24.247 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:14.342 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:24.288 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:14.343 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:24.283 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:29.215 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:39.161 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:29.263 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:39.208 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:29.289 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:39.235 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:29.311 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:39.250 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:29.312 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:39.251 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:29.317 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:39.263 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:29.320 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:39.265 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:29.329 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:39.269 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:29.340 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:39.286 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:34.272 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:44.218 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:34.277 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:44.222 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:34.294 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:44.239 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:34.323 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:44.262 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:34.325 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:44.264 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:34.325 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:44.271 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:34.329 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:44.275 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:34.342 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:44.288 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:34.349 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:44.289 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:49.316 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:59.261 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:49.321 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:59.266 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:49.338 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:59.284 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:49.368 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:59.307 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:49.369 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:59.315 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:49.370 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:59.310 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:49.371 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:59.317 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:49.386 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:59.325 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:49.389 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:59.335 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:54.330 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:04.276 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:54.334 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:04.280 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:54.348 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:04.293 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:54.377 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:04.316 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:54.380 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:04.319 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:54.382 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:04.327 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:54.384 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:04.330 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:54.390 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:04.330 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:26:54.397 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:04.342 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:09.369 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:19.314 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:09.371 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:19.317 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:09.390 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:19.335 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:09.419 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:19.358 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:09.421 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:19.361 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:09.424 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:19.370 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:09.427 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:19.373 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:09.437 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:19.376 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:09.444 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:19.389 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:14.384 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:24.329 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:14.385 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:24.331 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:14.399 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:24.344 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:14.431 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:24.371 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:14.432 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:24.378 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:14.432 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:24.372 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:14.435 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:24.381 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:14.443 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:24.383 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:14.448 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:24.394 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:29.424 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:39.369 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:29.429 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:39.374 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:29.447 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:39.392 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:29.477 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:39.416 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:29.478 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:39.417 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:29.478 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:39.424 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:29.480 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:39.426 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:29.494 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:39.434 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:29.498 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:39.443 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:32.280 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:27:32.272+0000 10a53640 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.192:6822 osd.6 since back 2023-12-15T17:26:06.166704+0000 front 2023-12-15T17:26:06.165149+0000 (oldest deadline 2023-12-15T17:27:31.463629+0000) 2023-12-15T17:27:33.244 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:27:33.242+0000 10a53640 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.192:6822 osd.6 since back 2023-12-15T17:26:06.166704+0000 front 2023-12-15T17:26:06.165149+0000 (oldest deadline 2023-12-15T17:27:31.463629+0000) 2023-12-15T17:27:34.281 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:27:34.278+0000 10a53640 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.192:6822 osd.6 since back 2023-12-15T17:26:06.166704+0000 front 2023-12-15T17:26:06.165149+0000 (oldest deadline 2023-12-15T17:27:31.463629+0000) 2023-12-15T17:27:34.440 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:44.385 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:34.442 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:44.387 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:34.455 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:44.400 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:34.485 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:44.431 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:34.488 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:44.427 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:34.489 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:44.429 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:34.490 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:44.435 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:34.500 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:44.439 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:34.502 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:44.448 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:35.269 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:27:35.268+0000 10a53640 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.192:6822 osd.6 since back 2023-12-15T17:26:06.166704+0000 front 2023-12-15T17:26:06.165149+0000 (oldest deadline 2023-12-15T17:27:31.463629+0000) 2023-12-15T17:27:36.293 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:27:36.291+0000 10a53640 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.192:6822 osd.6 since back 2023-12-15T17:26:06.166704+0000 front 2023-12-15T17:26:06.165149+0000 (oldest deadline 2023-12-15T17:27:31.463629+0000) 2023-12-15T17:27:37.300 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:27:37.298+0000 10a53640 -1 osd.7 926 heartbeat_check: no reply from 172.21.15.192:6822 osd.6 since back 2023-12-15T17:26:06.166704+0000 front 2023-12-15T17:26:06.165149+0000 (oldest deadline 2023-12-15T17:27:31.463629+0000) 2023-12-15T17:27:49.501 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:59.445 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:49.501 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:59.440 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:49.502 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:59.448 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:49.505 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:59.444 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:49.521 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:59.460 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:49.522 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:59.467 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:49.556 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:59.501 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:49.558 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:59.503 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:49.575 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:59.520 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:54.509 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:04.454 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:54.512 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:04.458 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:54.529 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:04.475 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:54.558 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:04.497 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:54.559 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:04.499 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:54.563 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:04.509 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:54.565 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:04.510 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:54.570 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:04.510 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:27:54.578 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:04.523 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:04.126 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:14.071 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:04.127 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:14.072 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:04.143 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:14.088 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:04.177 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:14.116 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:04.178 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:14.117 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:04.179 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:14.125 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:04.182 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:14.128 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:04.198 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:14.137 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:04.203 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:14.149 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:09.529 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:19.475 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:09.580 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:19.525 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:09.585 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:19.524 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:09.591 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:19.531 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:09.602 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:19.541 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:09.613 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:19.557 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:09.638 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:19.583 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:09.641 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:19.586 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:09.651 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:19.597 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:14.591 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:24.536 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:14.593 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:24.538 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:14.593 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:24.533 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:14.596 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:24.536 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:14.610 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:24.556 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:14.612 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:24.551 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:14.642 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:24.587 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:14.645 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:24.591 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:14.657 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:24.603 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:16.785 INFO:tasks.ceph.osd.6:Stopped 2023-12-15T17:28:16.785 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-15T17:28:16.785 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:28:16.797 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:28:16.788+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86803) UID: 0 2023-12-15T17:28:16.797 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-12-15T17:28:16.789+0000 b47e640 -1 osd.7 926 *** Got signal Terminated *** 2023-12-15T17:28:29.630 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:39.575 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:29.633 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:39.573 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:29.634 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:39.579 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:29.636 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:39.576 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:29.652 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:39.598 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:29.670 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:39.609 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:29.682 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:39.627 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:29.688 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:39.634 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:29.707 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:39.652 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:34.644 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:44.589 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:34.649 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:44.594 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:34.661 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:44.607 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:34.696 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:44.641 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:34.697 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:44.636 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:34.699 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:44.638 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:34.700 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:44.646 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:34.713 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:44.651 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:34.715 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:44.660 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:49.689 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:59.634 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:49.693 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:59.638 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:49.711 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:59.656 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:49.737 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:59.676 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:49.740 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:59.679 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:49.743 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:59.688 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:49.745 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:59.691 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:49.756 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:59.695 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:49.763 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:59.709 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:54.702 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:04.648 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:54.707 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:04.652 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:54.723 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:04.668 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:54.752 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:04.697 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:54.752 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:04.691 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:54.754 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:04.694 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:54.756 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:04.702 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:54.765 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:04.705 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:28:54.770 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:04.715 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:09.713 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:19.658 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:09.761 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:19.706 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:09.779 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:19.724 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:09.810 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:19.749 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:09.810 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:19.755 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:09.812 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:19.757 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:09.813 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:19.753 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:09.829 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:19.768 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:09.830 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:19.775 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:14.766 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:24.711 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:14.771 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:24.716 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:14.783 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:24.728 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:14.816 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:24.762 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:14.817 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:24.756 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:14.819 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:24.759 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:14.820 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:24.765 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:14.830 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:24.770 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:14.832 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:24.778 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:29.816 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:39.761 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:29.818 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:39.764 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:29.840 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:39.785 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:29.862 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:39.801 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:29.865 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:39.804 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:29.872 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:39.817 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:29.873 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:39.819 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:29.883 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:39.822 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:29.891 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:39.837 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:34.823 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:44.768 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:34.829 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:44.773 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:34.842 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:44.788 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:34.871 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:44.810 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:34.874 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:44.820 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:34.874 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:44.814 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:34.876 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:44.821 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:34.885 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:44.824 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:34.889 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:44.834 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:49.871 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:59.816 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:49.876 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:59.821 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:49.895 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:59.840 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:49.924 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:59.863 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:49.930 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:59.869 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:49.938 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:59.883 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:49.941 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:59.887 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:49.946 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:59.885 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:49.958 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:59.904 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:54.842 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:04.788 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:54.888 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:04.828 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:54.890 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:04.836 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:54.892 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:04.831 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:54.905 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:04.845 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:54.907 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:04.853 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:54.942 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:04.887 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:54.943 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:04.888 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:29:54.956 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:04.901 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:09.931 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:19.876 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:09.935 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:19.880 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:09.961 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:19.905 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:09.982 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:19.922 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:09.983 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:19.923 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:09.991 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:19.936 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:09.995 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:19.941 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:09.999 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:19.938 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:10.012 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:19.958 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:14.942 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:24.887 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:14.945 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:24.891 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:14.960 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:24.906 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:14.991 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:24.930 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:14.993 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:24.938 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:14.994 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:24.933 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:14.999 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:24.945 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:15.004 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:24.944 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:15.013 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:24.959 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:30.011 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:39.956 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:30.012 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:39.958 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:30.034 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:39.979 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:30.053 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:39.992 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:30.056 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:39.995 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:30.071 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:40.016 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:30.071 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:40.011 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:30.073 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:40.019 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:30.093 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:40.039 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:35.024 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:44.969 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:35.028 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:44.974 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:35.042 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:44.988 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:35.074 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:45.020 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:35.075 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:45.015 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:35.078 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:45.017 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:35.081 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:45.026 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:35.088 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:45.028 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:35.094 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:45.039 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:50.066 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:00.011 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:50.070 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:00.016 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:50.089 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:00.034 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:50.116 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:00.055 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:50.119 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:00.064 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:50.119 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:00.058 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:50.124 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:00.069 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:50.135 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:00.074 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:50.146 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:00.092 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:55.123 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:05.068 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:55.125 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:05.064 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:55.126 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:05.072 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:55.136 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:05.082 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:55.170 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:05.115 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:55.171 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:05.111 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:55.173 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:05.118 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:55.182 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:05.121 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:30:55.185 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:05.131 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:10.183 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:20.128 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:10.186 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:20.132 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:10.210 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:20.155 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:10.238 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:20.177 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:10.239 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:20.179 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:10.242 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:20.187 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:10.245 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:20.190 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:10.255 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:20.194 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:10.269 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:20.214 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.193 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:25.138 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.195 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:25.140 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.208 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:25.154 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.242 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:25.181 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.243 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:25.182 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.246 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:25.192 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.248 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:25.193 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.252 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:25.192 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.257 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:25.203 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.839 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:25.778 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.840 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:25.779 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.841 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:25.786 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.842 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:25.787 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.851 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:25.791 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.857 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:25.802 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.862 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:25.807 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.863 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:25.808 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:15.874 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:25.819 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:25.215 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:35.160 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:25.217 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:35.162 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:25.234 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:35.179 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:25.263 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:35.202 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:25.264 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:35.209 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:25.264 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:35.203 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:25.267 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:35.213 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:25.279 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:35.218 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:25.290 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:35.235 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:35.238 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:45.183 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:35.241 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:45.186 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:35.257 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:45.202 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:35.289 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:45.228 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:35.290 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:45.235 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:35.291 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:45.230 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:35.291 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:45.237 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:35.304 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:45.250 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:35.305 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:45.244 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:45.266 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:55.211 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:45.271 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:55.217 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:45.286 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:55.232 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:45.314 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:55.253 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:45.317 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:55.262 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:45.319 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:55.258 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:45.320 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:55.265 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:45.333 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:55.272 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:45.337 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:55.282 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:55.296 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:05.241 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:55.297 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:05.243 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:55.313 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:05.258 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:55.344 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:05.289 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:55.345 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:05.284 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:55.345 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:05.291 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:55.347 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:05.286 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:55.359 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:05.298 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:31:55.361 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:05.307 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:05.326 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:15.271 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:05.327 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:15.273 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:05.340 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:15.286 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:05.373 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:15.312 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:05.375 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:15.320 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:05.375 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:15.314 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:05.376 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:15.321 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:05.387 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:15.327 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:05.388 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:15.333 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:15.350 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:25.295 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:15.353 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:25.298 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:15.374 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:25.319 86427== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:15.398 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:25.337 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:15.400 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:25.340 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:15.406 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:25.352 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:15.409 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:25.354 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:15.414 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:25.353 86170== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:15.421 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:25.367 86430== Warning: unimplemented fcntl command: 1036 2023-12-15T17:32:20.842 INFO:tasks.ceph.osd.7:Stopped 2023-12-15T17:32:20.843 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-15T17:32:20.843 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-15T17:32:20.843 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:32:20.867 INFO:tasks.ceph.mgr.y:Stopped 2023-12-15T17:32:20.867 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-15T17:32:20.868 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:32:20.869 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:32:20.868+0000 7f70ac029640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 86168) UID: 0 2023-12-15T17:32:20.870 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-12-15T17:32:20.868+0000 7f70ac029640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-15T17:32:20.899 INFO:tasks.ceph.mgr.x:Stopped 2023-12-15T17:32:20.899 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-15T17:32:20.899 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-15T17:32:20.900 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:32:20.923 INFO:tasks.ceph.mon.a.smithi069.stderr:2023-12-15T17:32:20.912+0000 9f57640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86393) UID: 0 2023-12-15T17:32:20.923 INFO:tasks.ceph.mon.a.smithi069.stderr:2023-12-15T17:32:20.913+0000 9f57640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-15T17:32:21.554 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:31.499 86427== 2023-12-15T17:32:21.554 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:31.499 86427== Exit program on first error (--exit-on-first-error=yes) 2023-12-15T17:32:21.572 INFO:tasks.ceph.mon.a.smithi069.stderr:daemon-helper: command failed with exit status 42 2023-12-15T17:32:24.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-12-15T17:32:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-15T17:32:27.002 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-15T17:32:27.003 INFO:tasks.ceph.mon.a:Stopped 2023-12-15T17:32:27.003 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-15T17:32:27.003 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:32:27.018 INFO:tasks.ceph.mon.c.smithi069.stderr:2023-12-15T17:32:27.007+0000 9f57640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86395) UID: 0 2023-12-15T17:32:27.018 INFO:tasks.ceph.mon.c.smithi069.stderr:2023-12-15T17:32:27.008+0000 9f57640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-15T17:32:27.586 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:37.531 86430== 2023-12-15T17:32:27.586 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:37.531 86430== Exit program on first error (--exit-on-first-error=yes) 2023-12-15T17:32:27.602 INFO:tasks.ceph.mon.c.smithi069.stderr:daemon-helper: command failed with exit status 42 2023-12-15T17:32:29.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-15T17:32:29.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-12-15T17:32:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-15T17:32:33.105 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-15T17:32:33.105 INFO:tasks.ceph.mon.c:Stopped 2023-12-15T17:32:33.105 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-15T17:32:33.106 INFO:teuthology.orchestra.run:waiting for 300 2023-12-15T17:32:33.121 INFO:tasks.ceph.mon.b.smithi192.stderr:2023-12-15T17:32:33.108+0000 9f57640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 86166) UID: 0 2023-12-15T17:32:33.121 INFO:tasks.ceph.mon.b.smithi192.stderr:2023-12-15T17:32:33.110+0000 9f57640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-12-15T17:32:34.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-15T17:33:40.227 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:50.166 86170== 2023-12-15T17:33:40.227 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:50.166 86170== Exit program on first error (--exit-on-first-error=yes) 2023-12-15T17:33:40.243 INFO:tasks.ceph.mon.b.smithi192.stderr:daemon-helper: command failed with exit status 42 2023-12-15T17:33:40.250 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-15T17:33:40.250 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-15T17:33:40.250 INFO:tasks.ceph.mon.b:Stopped 2023-12-15T17:33:40.251 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-15T17:33:40.251 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-12-15T17:33:40.358 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi069.front.sepia.ceph.com 2023-12-15T17:33:40.358 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-15T17:33:40.422 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi069.front.sepia.ceph.com 2023-12-15T17:33:40.422 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-15T17:33:40.489 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi069.front.sepia.ceph.com 2023-12-15T17:33:40.489 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-15T17:33:40.546 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi069.front.sepia.ceph.com 2023-12-15T17:33:40.546 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-15T17:33:40.640 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi192.front.sepia.ceph.com 2023-12-15T17:33:40.640 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-15T17:33:40.718 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi192.front.sepia.ceph.com 2023-12-15T17:33:40.718 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-15T17:33:40.791 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi192.front.sepia.ceph.com 2023-12-15T17:33:40.791 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-15T17:33:40.868 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi192.front.sepia.ceph.com 2023-12-15T17:33:40.868 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-15T17:33:40.974 INFO:tasks.ceph:Archiving mon data... 2023-12-15T17:33:40.975 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262/data/mon.a.tgz 2023-12-15T17:33:40.976 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-12-15T17:33:40.992 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.TwNVq703u4 2023-12-15T17:33:40.993 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.TwNVq703u4 2023-12-15T17:33:41.145 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.TwNVq703u4 2023-12-15T17:33:41.229 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.TwNVq703u4 is 414KB 2023-12-15T17:33:41.325 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.TwNVq703u4 2023-12-15T17:33:41.339 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262/data/mon.c.tgz 2023-12-15T17:33:41.339 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-12-15T17:33:41.392 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.24WWqFLc6f 2023-12-15T17:33:41.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.24WWqFLc6f 2023-12-15T17:33:41.531 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.24WWqFLc6f 2023-12-15T17:33:41.616 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.24WWqFLc6f is 415KB 2023-12-15T17:33:41.713 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.24WWqFLc6f 2023-12-15T17:33:41.728 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262/data/mon.b.tgz 2023-12-15T17:33:41.728 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2023-12-15T17:33:41.743 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.TKUAgk6Fgy 2023-12-15T17:33:41.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.TKUAgk6Fgy 2023-12-15T17:33:41.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.TKUAgk6Fgy 2023-12-15T17:33:41.978 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.TKUAgk6Fgy is 414KB 2023-12-15T17:33:42.075 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.TKUAgk6Fgy 2023-12-15T17:33:42.089 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-15T17:33:42.089 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-15T17:33:42.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-15T17:33:42.476 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-15T17:33:42.740 INFO:tasks.ceph:Archiving crash dumps... 2023-12-15T17:33:42.741 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/crash to /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262/remote/smithi069/crash 2023-12-15T17:33:42.742 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-15T17:33:42.769 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/crash to /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262/remote/smithi192/crash 2023-12-15T17:33:42.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-15T17:33:42.796 INFO:tasks.ceph:Compressing logs... 2023-12-15T17:33:42.796 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-15T17:33:42.815 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86378.log 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86704.log 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.86378.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.86704.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86378.log.gz 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/valgrind/osd.7.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86704.log.gz 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/valgrind/osd.5.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr: 63.8% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-12-15T17:33:42.880 INFO:teuthology.orchestra.run.smithi192.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86418.log 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86549.log 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.86418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86338.log 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.86549.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86418.log.gz 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.86338.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.7.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86549.log.gz 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86589.log 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.5.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86338.log.gz 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-15T17:33:42.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.86589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86498.log 2023-12-15T17:33:42.882 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 88.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.86458.log 2023-12-15T17:33:42.882 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86589.log.gz 2023-12-15T17:33:42.882 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.86498.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-15T17:33:42.882 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.86458.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86498.log.gz 2023-12-15T17:33:42.882 INFO:teuthology.orchestra.run.smithi192.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.86458.log.gz 2023-12-15T17:33:42.882 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-15T17:33:42.882 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 92.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-15T17:33:42.882 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-15T17:33:42.889 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91407.log 2023-12-15T17:33:42.889 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88780.log 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92790.log 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91407.log: /var/log/ceph/ceph-client.admin.88780.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92790.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91407.log.gz 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110868.log 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88780.log.gz 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91457.log 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.110868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92802.log 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118224.log 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110868.log.gz 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92802.log: gzip 87.4% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.91457.log.gz -- 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr: /var/log/ceph/ceph-client.admin.90479.log 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92611.log 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.118224.log: /var/log/ceph/ceph-client.admin.90479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90979.log 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90751.log 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr: 91.2% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90479.log.gz 2023-12-15T17:33:42.890 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90979.log: -- replaced with /var/log/ceph/ceph-client.admin.92802.log.gz 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130698.log 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90751.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90979.log.gz 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92470.log 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.130698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92680.log 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90751.log.gz/var/log/ceph/ceph-client.admin.92470.log: 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92392.log 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.92790.log.gz 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90746.log 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92680.log: /var/log/ceph/ceph-client.admin.92392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86435.log 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.118224.log.gz/var/log/ceph/ceph-client.admin.90746.log: 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80583.log 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90746.log.gz/var/log/ceph/ceph-client.admin.86435.log: 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92083.log 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.tmp-client.admin.80583.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80583.log.gz 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.92680.log.gz 91.2% -- replaced with /var/log/ceph/ceph-client.admin.86435.log.gz 2023-12-15T17:33:42.891 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr:gzip 90.4% -- replaced with /var/log/ceph/ceph-client.admin.130698.log.gz 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92083.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr: 74.7%/var/log/ceph/valgrind/osd.2.log: -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose/var/log/ceph/valgrind/osd.0.log: -- /var/log/ceph/valgrind/mon.a.log 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1% 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/osd.1.log: -- replaced with /var/log/ceph/ceph-client.admin.92083.log.gz 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gzgzip 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91497.log 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/mon.a.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.92392.log.gz 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/osd.3.log: 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-12-15T17:33:42.892 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91306.log 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91365.log 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158567.log 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91497.log.gz 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92227.log 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91306.log.gz 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5/var/log/ceph/ceph-client.admin.158567.log: --verbose -- /var/log/ceph/ceph-client.admin.130693.log 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8%/var/log/ceph/ceph-client.admin.92227.log: -- replaced with /var/log/ceph/ceph-client.admin.91365.log.gz 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158357.log 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.158567.log.gz 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.130693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92602.log 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92126.log 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92602.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92479.log 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.158357.log.gz 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158189.log 2023-12-15T17:33:42.893 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92479.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92470.log.gz 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91623.log 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158189.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.130693.log.gz 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.92602.log.gz 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.158189.log.gz 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86910.log 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.91623.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.157979.log 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.86910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92458.log 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91623.log.gz/var/log/ceph/ceph-client.admin.157979.log: 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86910.log.gz 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90752.log 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92458.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.157979.log.gz 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106652.log 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92210.log 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1%/var/log/ceph/ceph-client.admin.106652.log: -- replaced with /var/log/ceph/ceph-client.admin.92227.log.gz 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90752.log.gz 2023-12-15T17:33:42.894 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130695.log 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158483.log 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.92126.log.gz 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.130695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90988.log 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158483.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.158483.log.gz 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.106652.log.gz 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90750.log 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90988.log: gzip -5 91.6% --verbose -- /var/log/ceph/ceph-client.admin.91067.log 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92479.log.gz 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158651.log 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90988.log.gz 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91067.log: 90.7% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90750.log.gz 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.130695.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158777.log 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158651.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91067.log.gz 2023-12-15T17:33:42.895 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90333.log 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.158651.log.gz 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92761.log 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110943.log 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6%/var/log/ceph/ceph-client.admin.92761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158609.log 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.158777.log.gz 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90333.log.gzgzip 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5%/var/log/ceph/ceph-client.admin.110943.log: -- replaced with /var/log/ceph/ceph-client.admin.92761.log.gz 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158063.log 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158231.log 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.92458.log.gz 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158609.log.gz 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110944.log 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158063.log: /var/log/ceph/ceph-client.admin.158231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90437.log 2023-12-15T17:33:42.896 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.110944.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.158063.log.gz 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158231.log.gz 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92047.log 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110944.log.gz 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92369.log 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.110943.log.gz 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90971.log 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90437.log.gz 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.92369.log: -- /var/log/ceph/ceph-client.admin.92792.log 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130696.log 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90971.log.gz 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92094.log 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.130696.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92792.log.gz 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.130696.log.gz 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.92094.log: -- /var/log/ceph/ceph-client.admin.158735.log 2023-12-15T17:33:42.897 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.1.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92047.log.gz 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.92369.log.gz 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92452.log 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106653.log 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92452.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158735.log.gz 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90286.log 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.106653.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.92452.log.gz 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.106653.log.gz 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90569.log 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91707.log 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90569.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144871.log 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90286.log.gz 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91707.log: gzip -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90569.log.gz 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.90744.log 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.144871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92319.log 2023-12-15T17:33:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90744.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91707.log.gz 2023-12-15T17:33:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92709.log 2023-12-15T17:33:42.899 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90744.log.gz 2023-12-15T17:33:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92319.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-15T17:33:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92709.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.144871.log.gz 2023-12-15T17:33:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92056.log 2023-12-15T17:33:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.3.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.92709.log.gz 2023-12-15T17:33:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158693.log 2023-12-15T17:33:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92056.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92611.log.gz 2023-12-15T17:33:42.902 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92009.log 2023-12-15T17:33:42.902 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158693.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158693.log.gz 2023-12-15T17:33:42.902 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144870.log 2023-12-15T17:33:42.902 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92009.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.92056.log.gz 2023-12-15T17:33:42.902 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86752.log 2023-12-15T17:33:42.902 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.144870.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.144870.log.gz 2023-12-15T17:33:42.902 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91113.log 2023-12-15T17:33:42.903 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.86752.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.86752.log.gz 2023-12-15T17:33:42.904 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90737.log 2023-12-15T17:33:42.904 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91113.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91113.log.gz 2023-12-15T17:33:42.905 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90393.log 2023-12-15T17:33:42.905 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90737.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90737.log.gz 2023-12-15T17:33:42.905 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158819.log 2023-12-15T17:33:42.906 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90393.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90393.log.gz 2023-12-15T17:33:42.907 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92185.log 2023-12-15T17:33:42.908 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158819.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158819.log.gz 2023-12-15T17:33:42.908 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90523.log 2023-12-15T17:33:42.920 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118226.log 2023-12-15T17:33:42.921 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90523.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90523.log.gz 2023-12-15T17:33:42.925 INFO:teuthology.orchestra.run.smithi069.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.92009.log.gz 2023-12-15T17:33:42.925 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158525.log 2023-12-15T17:33:42.929 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.118226.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.118226.log.gz 2023-12-15T17:33:42.929 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-15T17:33:42.929 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158525.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.158525.log.gz 2023-12-15T17:33:42.930 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157937.log 2023-12-15T17:33:42.931 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mgr.y.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.92094.log.gz 2023-12-15T17:33:42.931 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91071.log 2023-12-15T17:33:42.933 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.157937.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.157937.log.gz 2023-12-15T17:33:42.933 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118225.log 2023-12-15T17:33:42.934 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91071.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91071.log.gz 2023-12-15T17:33:42.935 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92801.log 2023-12-15T17:33:42.937 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.118225.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.118225.log.gz 2023-12-15T17:33:42.937 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158859.log 2023-12-15T17:33:42.942 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92801.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.92801.log.gz 2023-12-15T17:33:42.942 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158315.log 2023-12-15T17:33:42.943 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158859.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158859.log.gz 2023-12-15T17:33:42.943 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144872.log 2023-12-15T17:33:42.945 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158315.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158315.log.gz 2023-12-15T17:33:42.945 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-15T17:33:42.947 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.144872.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.144872.log.gz 2023-12-15T17:33:42.952 INFO:teuthology.orchestra.run.smithi069.stderr: 94.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-15T17:33:42.953 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91068.log 2023-12-15T17:33:42.953 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110946.log 2023-12-15T17:33:42.954 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91068.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91068.log.gz 2023-12-15T17:33:42.975 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158105.log 2023-12-15T17:33:42.977 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.110946.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.110946.log.gz 2023-12-15T17:33:42.983 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92026.log 2023-12-15T17:33:42.984 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158105.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.158105.log.gz 2023-12-15T17:33:42.986 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158441.log 2023-12-15T17:33:42.996 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92112.log 2023-12-15T17:33:42.997 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158441.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.158441.log.gz 2023-12-15T17:33:43.000 INFO:teuthology.orchestra.run.smithi069.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.92185.log.gz 2023-12-15T17:33:43.012 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-15T17:33:43.017 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92112.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92112.log.gz 2023-12-15T17:33:43.018 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91527.log 2023-12-15T17:33:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.log: 92.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90755.log 2023-12-15T17:33:43.031 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91527.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91527.log.gz 2023-12-15T17:33:43.031 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2023-12-15T17:33:43.031 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91307.log 2023-12-15T17:33:43.032 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90755.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90755.log.gz 2023-12-15T17:33:43.032 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-15T17:33:43.033 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91307.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91307.log.gz 2023-12-15T17:33:43.034 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92583.log 2023-12-15T17:33:43.047 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158147.log 2023-12-15T17:33:43.051 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92583.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92026.log.gz 2023-12-15T17:33:43.054 INFO:teuthology.orchestra.run.smithi069.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.92583.log.gz 2023-12-15T17:33:43.055 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130699.log 2023-12-15T17:33:43.055 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158147.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158147.log.gz 2023-12-15T17:33:43.056 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86779.log 2023-12-15T17:33:43.059 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.130699.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.130699.log.gz 2023-12-15T17:33:43.059 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158273.log 2023-12-15T17:33:43.060 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.86779.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86779.log.gz 2023-12-15T17:33:43.061 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92831.log 2023-12-15T17:33:43.061 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158273.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158273.log.gz 2023-12-15T17:33:43.062 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92507.log 2023-12-15T17:33:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92831.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.92831.log.gz 2023-12-15T17:33:43.067 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110866.log 2023-12-15T17:33:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92507.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92507.log.gz 2023-12-15T17:33:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91427.log 2023-12-15T17:33:43.079 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.110866.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.110866.log.gz 2023-12-15T17:33:43.080 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90748.log 2023-12-15T17:33:43.080 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91427.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91427.log.gz 2023-12-15T17:33:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91665.log 2023-12-15T17:33:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90748.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90748.log.gz 2023-12-15T17:33:43.082 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90049.log 2023-12-15T17:33:43.083 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91665.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91665.log.gz 2023-12-15T17:33:43.083 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91052.log 2023-12-15T17:33:43.085 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.90049.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90049.log.gz 2023-12-15T17:33:43.085 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158021.log 2023-12-15T17:33:43.086 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.91052.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91052.log.gz 2023-12-15T17:33:43.086 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158399.log 2023-12-15T17:33:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158021.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.158021.log.gz 2023-12-15T17:33:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106654.log 2023-12-15T17:33:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.158399.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158399.log.gz 2023-12-15T17:33:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87771.log 2023-12-15T17:33:43.093 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.106654.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.106654.log.gz 2023-12-15T17:33:43.093 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92832.log 2023-12-15T17:33:43.094 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.87771.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87771.log.gz 2023-12-15T17:33:43.095 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92546.log 2023-12-15T17:33:43.096 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92832.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.92832.log.gz 2023-12-15T17:33:43.096 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-15T17:33:43.120 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.92546.log: 91.2%/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.admin.92546.log.gz 2023-12-15T17:33:43.220 INFO:teuthology.orchestra.run.smithi192.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-15T17:33:43.406 INFO:teuthology.orchestra.run.smithi069.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.92319.log.gz 2023-12-15T17:33:44.200 INFO:teuthology.orchestra.run.smithi069.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92210.log.gz 2023-12-15T17:33:44.682 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-15T17:33:44.710 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-15T17:33:45.485 INFO:teuthology.orchestra.run.smithi069.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-15T17:33:46.494 INFO:teuthology.orchestra.run.smithi192.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-15T17:33:46.561 INFO:teuthology.orchestra.run.smithi069.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-15T17:33:46.633 INFO:teuthology.orchestra.run.smithi192.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-15T17:33:46.879 INFO:teuthology.orchestra.run.smithi192.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-15T17:33:47.095 INFO:teuthology.orchestra.run.smithi069.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-15T17:33:47.136 INFO:teuthology.orchestra.run.smithi192.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-15T17:33:47.137 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-12-15T17:33:47.138 INFO:teuthology.orchestra.run.smithi192.stderr:real 0m4.286s 2023-12-15T17:33:47.138 INFO:teuthology.orchestra.run.smithi192.stderr:user 0m17.271s 2023-12-15T17:33:47.138 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m0.592s 2023-12-15T17:33:47.347 INFO:teuthology.orchestra.run.smithi069.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-15T17:33:47.621 INFO:teuthology.orchestra.run.smithi069.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-15T17:33:47.623 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-12-15T17:33:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m4.796s 2023-12-15T17:33:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m22.792s 2023-12-15T17:33:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m0.815s 2023-12-15T17:33:47.623 INFO:tasks.ceph:Archiving logs... 2023-12-15T17:33:47.624 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262/remote/smithi069/log 2023-12-15T17:33:47.624 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-15T17:33:50.789 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262/remote/smithi192/log 2023-12-15T17:33:50.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-15T17:33:53.281 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-15T17:33:53.282 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-15T17:33:53.291 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-15T17:33:53.292 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-15T17:33:53.292 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-15T17:33:53.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-15T17:33:53.380 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-15T17:33:53.389 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-15T17:33:53.389 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-15T17:33:53.392 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-15T17:33:53.406 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2023-12-15T17:33:53.408 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2023-12-15T17:33:53.411 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-15T17:33:53.411 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2023-12-15T17:33:53.411 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 -563us[ -549us] +/- 93ms 2023-12-15T17:33:53.411 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 -457us[ -443us] +/- 58ms 2023-12-15T17:33:53.411 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 +1170us[+1184us] +/- 84ms 2023-12-15T17:33:53.411 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T17:33:53.414 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-15T17:33:53.414 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2023-12-15T17:33:53.414 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 -569us[ -569us] +/- 93ms 2023-12-15T17:33:53.414 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -456us[ -473us] +/- 58ms 2023-12-15T17:33:53.414 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +1168us[+1168us] +/- 84ms 2023-12-15T17:33:53.415 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T17:33:53.415 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-15T17:33:53.425 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-15T17:33:53.426 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-15T17:33:53.448 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-15T17:33:53.552 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-15T17:33:53.651 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-15T17:33:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T17:33:53.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-15T17:33:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T17:33:53.717 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-15T17:33:53.726 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-15T17:33:53.763 INFO:teuthology.task.internal:Duration was 3007.782619 seconds 2023-12-15T17:33:53.764 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-15T17:33:53.796 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-15T17:33:53.796 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-15T17:33:53.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-15T17:33:53.839 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-15T17:33:53.843 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-15T17:33:54.310 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-15T17:33:54.310 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2023-12-15T17:33:54.310 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-15T17:33:54.330 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2023-12-15T17:33:54.330 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-15T17:33:54.351 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-15T17:33:54.352 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-15T17:33:54.373 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-15T17:33:54.420 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-15T17:33:54.420 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-15T17:33:54.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-15T17:33:54.556 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-15T17:33:54.567 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-15T17:33:54.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-15T17:33:54.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-15T17:33:54.621 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-15T17:33:54.693 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-15T17:33:54.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-15T17:33:54.719 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2023-12-15T17:33:54.721 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2023-12-15T17:33:54.760 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-15T17:33:54.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T17:33:54.790 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-15T17:33:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T17:33:54.814 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-15T17:33:54.825 INFO:teuthology.task.internal:Transferring archived files... 2023-12-15T17:33:54.826 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262/remote/smithi069 2023-12-15T17:33:54.826 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-15T17:33:54.889 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262/remote/smithi192 2023-12-15T17:33:54.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-15T17:33:54.927 INFO:teuthology.task.internal:Removing archive directory... 2023-12-15T17:33:54.928 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-15T17:33:54.930 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-15T17:33:54.976 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-15T17:33:54.988 INFO:teuthology.task.internal:Not uploading archives. 2023-12-15T17:33:54.989 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-15T17:33:55.006 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-15T17:33:55.006 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-15T17:33:55.008 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-15T17:33:55.020 INFO:teuthology.orchestra.run.smithi069.stdout: 265245 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 15 17:33 /home/ubuntu/cephtest 2023-12-15T17:33:55.030 INFO:teuthology.orchestra.run.smithi192.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 15 17:33 /home/ubuntu/cephtest 2023-12-15T17:33:55.031 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-15T17:33:55.041 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-15T17:33:55.073 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-15T17:33:55.073 INFO:teuthology.nuke:Checking targets against current locks 2023-12-15T17:33:55.099 DEBUG:teuthology.nuke:shortname: smithi069 2023-12-15T17:33:55.099 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-15T17:33:55.122 DEBUG:teuthology.nuke:shortname: smithi192 2023-12-15T17:33:55.122 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-15T17:33:55.125 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-15 16:35:31.136718', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC2/eNabySf0IuZlE+BHIHvM/dceKYYIFmCN2lkbZT9YqRA6QeS/zPOLyRDI83a0zpFobrC9ZbJzDK85o4PM83k='} 2023-12-15T17:33:55.148 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2023-12-15_16:16:51-rados-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7493262', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-15 16:35:31.134701', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJSe1/+NcQykE1SfbIJYGnBvdauZOKvJs4f8q5eTJzeX4G6JG/f5bVREqsRWL1z0Szf90F/G1grP+vyeGpIPDBk='} 2023-12-15T17:33:55.171 INFO:teuthology.orchestra.console.smithi069:Power off 2023-12-15T17:33:55.172 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-15T17:33:55.193 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2023-12-15T17:33:55.194 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-15T17:33:55.210 INFO:teuthology.orchestra.console.smithi192:Power off 2023-12-15T17:33:55.210 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-15T17:33:55.221 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-12-15T17:33:55.231 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2023-12-15T17:33:55.231 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-15T17:33:55.254 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-12-15T17:33:59.223 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-15T17:33:59.255 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-15T17:33:59.332 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-12-15T17:33:59.365 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-12-15T17:34:03.333 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-15T17:34:03.366 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-15T17:34:03.444 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2023-12-15T17:34:03.476 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-12-15T17:34:03.545 INFO:teuthology.orchestra.console.smithi069:Power off completed 2023-12-15T17:34:07.478 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-15T17:34:07.590 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2023-12-15T17:34:07.691 INFO:teuthology.orchestra.console.smithi192:Power off completed 2023-12-15T17:34:07.797 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 3007.7826194763184 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate' flavor: default owner: scheduled_lflores@teuthology status: fail success: false 2023-12-15T17:34:07.798 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-15T17:34:07.870 INFO:teuthology.run:FAIL